Patchwork [6,of,6] run-tests.py: inherit PYTHONHASHSEED from environment if set

login
register
mail settings
Submitter Mads Kiilerich
Date Jan. 20, 2013, 1:18 p.m.
Message ID <f62e594f1b67ea617c8d.1358687889@localhost6.localdomain6>
Download mbox | patch
Permalink /patch/692/
State Accepted
Commit f0124a65e8f8d2498d76e3547cbb2fed866709cb
Headers show

Comments

Mads Kiilerich - Jan. 20, 2013, 1:18 p.m.
# HG changeset patch
# User Mads Kiilerich <mads@kiilerich.com>
# Date 1358687872 -3600
# Branch stable
# Node ID f62e594f1b67ea617c8d727834ee8c9895efe4ef
# Parent  a0e2208420d688baa24bb52d9064e139a9dcc885
run-tests.py: inherit PYTHONHASHSEED from environment if set

This makes it possible to fix the seed by using for instance
  PYTHONHASHSEED=7 ./run-tests.py ...

This can be very convenient when trying to debug problems that are influenced
by hash values. Try different seed values until you find one that triggers the
bad behaviour and then keep that while debugging.

It could also to some extent make sense for buildbots to set (different) fixed
values to make sure the test execution is stable.
Matt Mackall - Jan. 20, 2013, 9:24 p.m.
On Sun, 2013-01-20 at 14:18 +0100, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich <mads@kiilerich.com>
> # Date 1358687872 -3600
> # Branch stable
> # Node ID f62e594f1b67ea617c8d727834ee8c9895efe4ef
> # Parent  a0e2208420d688baa24bb52d9064e139a9dcc885
> run-tests.py: inherit PYTHONHASHSEED from environment if set
> 
> This makes it possible to fix the seed by using for instance
>   PYTHONHASHSEED=7 ./run-tests.py ...
> 
> This can be very convenient when trying to debug problems that are influenced
> by hash values. Try different seed values until you find one that triggers the
> bad behaviour and then keep that while debugging.
> 
> It could also to some extent make sense for buildbots to set (different) fixed
> values to make sure the test execution is stable.

I'm not sure precisely what you mean here. I think there's value in
running the buildbot with random seeds. However, it's also useful to be
able to rerun a test with the same seed a buildbot uses. As there's no
way to access the seed used for PYTHONHASHSEED=random, perhaps the
buildbot or run-test should generate a seed and log it.

> diff --git a/tests/run-tests.py b/tests/run-tests.py
> --- a/tests/run-tests.py
> +++ b/tests/run-tests.py
> @@ -1252,7 +1252,7 @@ def main():
>      os.environ['no_proxy'] = ''
>      os.environ['NO_PROXY'] = ''
>      os.environ['TERM'] = 'xterm'
> -    os.environ['PYTHONHASHSEED'] = 'random'
> +    os.environ['PYTHONHASHSEED'] = os.environ.get('PYTHONHASHSEED', 'random')
>  
>      # unset env related to hooks
>      for k in os.environ.keys():
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@selenic.com
> http://selenic.com/mailman/listinfo/mercurial-devel
Mads Kiilerich - Jan. 20, 2013, 10 p.m.
Matt Mackall wrote, On 01/20/2013 10:24 PM:
> On Sun, 2013-01-20 at 14:18 +0100, Mads Kiilerich wrote:
>> # HG changeset patch
>> # User Mads Kiilerich <mads@kiilerich.com>
>> # Date 1358687872 -3600
>> # Branch stable
>> # Node ID f62e594f1b67ea617c8d727834ee8c9895efe4ef
>> # Parent  a0e2208420d688baa24bb52d9064e139a9dcc885
>> run-tests.py: inherit PYTHONHASHSEED from environment if set
>>
>> This makes it possible to fix the seed by using for instance
>>    PYTHONHASHSEED=7 ./run-tests.py ...
>>
>> This can be very convenient when trying to debug problems that are influenced
>> by hash values. Try different seed values until you find one that triggers the
>> bad behaviour and then keep that while debugging.
>>
>> It could also to some extent make sense for buildbots to set (different) fixed
>> values to make sure the test execution is stable.
> I'm not sure precisely what you mean here. I think there's value in
> running the buildbot with random seeds. However, it's also useful to be
> able to rerun a test with the same seed a buildbot uses. As there's no
> way to access the seed used for PYTHONHASHSEED=random, perhaps the
> buildbot or run-test should generate a seed and log it.

Agreed.

My thought was that every buildbot using a different fixed "random" 
number also would give good coverage and perhaps be less confusing ... 
and make sure that all test runs are fully reproducible.

I guess either way could work ... but we should perhaps wait and see 
what kind of problems we end up having on the buildbot. Having a way to 
control it locally is however very convenient.

/Mads

(Mandatory references: http://dilbert.com/strips/comic/2001-10-25/ and 
http://xkcd.com/221/ )

>> diff --git a/tests/run-tests.py b/tests/run-tests.py
>> --- a/tests/run-tests.py
>> +++ b/tests/run-tests.py
>> @@ -1252,7 +1252,7 @@ def main():
>>       os.environ['no_proxy'] = ''
>>       os.environ['NO_PROXY'] = ''
>>       os.environ['TERM'] = 'xterm'
>> -    os.environ['PYTHONHASHSEED'] = 'random'
>> +    os.environ['PYTHONHASHSEED'] = os.environ.get('PYTHONHASHSEED', 'random')
>>   
>>       # unset env related to hooks
>>       for k in os.environ.keys():
>> _______________________________________________
>> Mercurial-devel mailing list
>> Mercurial-devel@selenic.com
>> http://selenic.com/mailman/listinfo/mercurial-devel
>
Matt Mackall - Jan. 20, 2013, 10:06 p.m.
On Sun, 2013-01-20 at 23:00 +0100, Mads Kiilerich wrote:
> Matt Mackall wrote, On 01/20/2013 10:24 PM:
> > On Sun, 2013-01-20 at 14:18 +0100, Mads Kiilerich wrote:
> >> # HG changeset patch
> >> # User Mads Kiilerich <mads@kiilerich.com>
> >> # Date 1358687872 -3600
> >> # Branch stable
> >> # Node ID f62e594f1b67ea617c8d727834ee8c9895efe4ef
> >> # Parent  a0e2208420d688baa24bb52d9064e139a9dcc885
> >> run-tests.py: inherit PYTHONHASHSEED from environment if set
> >>
> >> This makes it possible to fix the seed by using for instance
> >>    PYTHONHASHSEED=7 ./run-tests.py ...
> >>
> >> This can be very convenient when trying to debug problems that are influenced
> >> by hash values. Try different seed values until you find one that triggers the
> >> bad behaviour and then keep that while debugging.
> >>
> >> It could also to some extent make sense for buildbots to set (different) fixed
> >> values to make sure the test execution is stable.
> > I'm not sure precisely what you mean here. I think there's value in
> > running the buildbot with random seeds. However, it's also useful to be
> > able to rerun a test with the same seed a buildbot uses. As there's no
> > way to access the seed used for PYTHONHASHSEED=random, perhaps the
> > buildbot or run-test should generate a seed and log it.
> 
> Agreed.
> 
> My thought was that every buildbot using a different fixed "random" 
> number also would give good coverage and perhaps be less confusing ... 
> and make sure that all test runs are fully reproducible.

Then we're prone to associate failures that are hash-related with
platforms instead.

Patch

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -1252,7 +1252,7 @@  def main():
     os.environ['no_proxy'] = ''
     os.environ['NO_PROXY'] = ''
     os.environ['TERM'] = 'xterm'
-    os.environ['PYTHONHASHSEED'] = 'random'
+    os.environ['PYTHONHASHSEED'] = os.environ.get('PYTHONHASHSEED', 'random')
 
     # unset env related to hooks
     for k in os.environ.keys():