Patchwork [6,of,6] run-tests: introduce --random for running tests in random error

login
register
mail settings
Submitter Mads Kiilerich
Date April 17, 2013, 2:11 a.m.
Message ID <0dbf64735055e8dc2118.1366164666@xps>
Download mbox | patch
Permalink /patch/1375/
State Accepted, archived
Commit 3d265e0822d36332b0c995b22f58a90d841241ab
Headers show

Comments

Mads Kiilerich - April 17, 2013, 2:11 a.m.
# HG changeset patch
# User Mads Kiilerich <madski@unity3d.com>
# Date 1366162894 -7200
# Node ID 0dbf64735055e8dc2118ddea7eacb76118045235
# Parent  5dbf61fe581870e2c5b3a50c39dcc5f90cc161dc
run-tests: introduce --random for running tests in random error

This is convenient when investigating failures one by one to avoid running the
same fixed tests first every time.
Augie Fackler - April 17, 2013, 3:14 a.m.
On Apr 16, 2013, at 10:11 PM, Mads Kiilerich <mads@kiilerich.com> wrote:

> # HG changeset patch
> # User Mads Kiilerich <madski@unity3d.com>
> # Date 1366162894 -7200
> # Node ID 0dbf64735055e8dc2118ddea7eacb76118045235
> # Parent  5dbf61fe581870e2c5b3a50c39dcc5f90cc161dc
> run-tests: introduce --random for running tests in random error
> 
> This is convenient when investigating failures one by one to avoid running the
> same fixed tests first every time.

Huh? I'm not quite sure I understand the merit of this one.

> 
> diff --git a/tests/run-tests.py b/tests/run-tests.py
> --- a/tests/run-tests.py
> +++ b/tests/run-tests.py
> @@ -198,6 +198,8 @@ def parseargs():
>         help="enable Py3k warnings on Python 2.6+")
>     parser.add_option('--extra-config-opt', action="append",
>                       help='set the given config opt in the test hgrc')
> +    parser.add_option('--random', action="store_true",
> +                      help='run tests in random order')
> 
>     for option, (envvar, default) in defaults.items():
>         defaults[option] = type(default)(os.environ.get(envvar, default))
> @@ -1248,6 +1250,9 @@ def main():
> 
>     tests = args
> 
> +    if options.random:
> +        random.shuffle(tests)
> +
>     # Reset some environment variables to well-known values so that
>     # the tests produce repeatable output.
>     os.environ['LANG'] = os.environ['LC_ALL'] = os.environ['LANGUAGE'] = 'C'
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@selenic.com
> http://selenic.com/mailman/listinfo/mercurial-devel
Mads Kiilerich - April 17, 2013, 2:01 p.m.
On 04/17/2013 05:14 AM, Augie Fackler wrote:
> On Apr 16, 2013, at 10:11 PM, Mads Kiilerich <mads@kiilerich.com> wrote:
>
>> # HG changeset patch
>> # User Mads Kiilerich <madski@unity3d.com>
>> # Date 1366162894 -7200
>> # Node ID 0dbf64735055e8dc2118ddea7eacb76118045235
>> # Parent  5dbf61fe581870e2c5b3a50c39dcc5f90cc161dc
>> run-tests: introduce --random for running tests in random error
>>
>> This is convenient when investigating failures one by one to avoid running the
>> same fixed tests first every time.
> Huh? I'm not quite sure I understand the merit of this one.

I "needed" this when testing a change that caused lots of tests to fail. 
I stopped run-tests whenever a test failed and tweaked code or test 
until it passed. Then I started run-tests again waiting for it to fail 
next time ... and I got bored of watching it testing the 'a' tests first 
every time. Using a random order worked better for me. Also because it 
often is convenient to attack hard problems from different angles.

My use case could also benefit from run-tests creating .err files up 
front for all tests it would consider running ... and remove them when 
tests passed. Then I could resume with -r.

But yeah, not a big and obvious feature. I will keep it in my queue and 
use it when I need it ;-)

/Mads
moving on to solving the problem of having seen more endings of TV films 
than beginnings and having heard the beginnings of albums more often 
than the ends


>
>> diff --git a/tests/run-tests.py b/tests/run-tests.py
>> --- a/tests/run-tests.py
>> +++ b/tests/run-tests.py
>> @@ -198,6 +198,8 @@ def parseargs():
>>          help="enable Py3k warnings on Python 2.6+")
>>      parser.add_option('--extra-config-opt', action="append",
>>                        help='set the given config opt in the test hgrc')
>> +    parser.add_option('--random', action="store_true",
>> +                      help='run tests in random order')
>>
>>      for option, (envvar, default) in defaults.items():
>>          defaults[option] = type(default)(os.environ.get(envvar, default))
>> @@ -1248,6 +1250,9 @@ def main():
>>
>>      tests = args
>>
>> +    if options.random:
>> +        random.shuffle(tests)
>> +
>>      # Reset some environment variables to well-known values so that
>>      # the tests produce repeatable output.
>>      os.environ['LANG'] = os.environ['LC_ALL'] = os.environ['LANGUAGE'] = 'C'
>> _______________________________________________
>> Mercurial-devel mailing list
>> Mercurial-devel@selenic.com
>> http://selenic.com/mailman/listinfo/mercurial-devel
Matt Mackall - April 18, 2013, 5:43 a.m.
On Wed, 2013-04-17 at 04:11 +0200, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich <madski@unity3d.com>
> # Date 1366162894 -7200
> # Node ID 0dbf64735055e8dc2118ddea7eacb76118045235
> # Parent  5dbf61fe581870e2c5b3a50c39dcc5f90cc161dc
> run-tests: introduce --random for running tests in random error

Queued for default, thanks.

Patch

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -198,6 +198,8 @@  def parseargs():
         help="enable Py3k warnings on Python 2.6+")
     parser.add_option('--extra-config-opt', action="append",
                       help='set the given config opt in the test hgrc')
+    parser.add_option('--random', action="store_true",
+                      help='run tests in random order')
 
     for option, (envvar, default) in defaults.items():
         defaults[option] = type(default)(os.environ.get(envvar, default))
@@ -1248,6 +1250,9 @@  def main():
 
     tests = args
 
+    if options.random:
+        random.shuffle(tests)
+
     # Reset some environment variables to well-known values so that
     # the tests produce repeatable output.
     os.environ['LANG'] = os.environ['LC_ALL'] = os.environ['LANGUAGE'] = 'C'