Patchwork run-tests: add --slowtimeout and use it for slow tests

login
register
mail settings
Submitter timeless@mozdev.org
Date Nov. 25, 2015, 1:21 a.m.
Message ID <3e22d954076f42dedb7e.1448414495@waste.org>
Download mbox | patch
Permalink /patch/11646/
State Accepted
Delegated to: Yuya Nishihara
Headers show

Comments

timeless@mozdev.org - Nov. 25, 2015, 1:21 a.m.
# HG changeset patch
# User timeless <timeless@mozdev.org>
# Date 1448411945 0
#      Wed Nov 25 00:39:05 2015 +0000
# Node ID 3e22d954076f42dedb7e2391bc01e3640f2130b9
# Parent  df9b73d2d444ae82fe8d3fe6cf682a93b2c4a7ef
run-tests: add --slowtimeout and use it for slow tests
Yuya Nishihara - Nov. 28, 2015, 1:18 p.m.
On Tue, 24 Nov 2015 19:21:35 -0600, timeless wrote:
> # HG changeset patch
> # User timeless <timeless@mozdev.org>
> # Date 1448411945 0
> #      Wed Nov 25 00:39:05 2015 +0000
> # Node ID 3e22d954076f42dedb7e2391bc01e3640f2130b9
> # Parent  df9b73d2d444ae82fe8d3fe6cf682a93b2c4a7ef
> run-tests: add --slowtimeout and use it for slow tests

> @@ -469,6 +480,7 @@
>          self._keeptmpdir = keeptmpdir
>          self._debug = debug
>          self._timeout = timeout
> +        self._slowtimeout = slowtimeout
>          self._startport = startport
>          self._extraconfigopts = extraconfigopts or []
>          self._py3kwarnings = py3kwarnings
> @@ -922,7 +934,12 @@
>              print(stdout)
>              sys.exit(1)
>  
> -        return ret == 0
> +        if ret != 0:
> +            return False
> +
> +        if 'slow' in reqs:
> +            self._timeout = self._slowtimeout

Perhaps it won't work if there's a "#if slow" conditional.

> +timeouts
> +========
> +  $ cat > test-timeout.t <<EOF
> +  >   $ sleep 10
> +  >   $ echo pass
> +  >   pass
> +  > EOF
> +  > echo '#require slow' > test-slow-timeout.t
> +  > cat test-timeout.t >> test-slow-timeout.t
> +  $ run-tests.py --with-hg=`which hg` --timeout=1 --slowtimeout=20 test-timeout.t test-slow-timeout.t
> +  s
> +  ERROR: test-timeout.t output changed
> +  !
> +  Skipped test-slow-timeout.t: skipped
> +  Failed test-timeout.t: timed out
> +  # Ran 1 tests, 1 skipped, 0 warned, 1 failed.
> +  python hash seed: * (glob)
> +  [1]
> +  $ run-tests.py --with-hg=`which hg` --timeout=1 --slowtimeout=20 test-timeout.t test-slow-timeout.t --allow-slow-tests
> +  .
> +  ERROR: test-timeout.t output changed
> +  !
> +  Failed test-timeout.t: timed out
> +  # Ran 2 tests, 0 skipped, 0 warned, 1 failed.
> +  python hash seed: * (glob)
> +  [1]
> +  $ rm test-timeout.t test-slow-timeout.t

and I think two "sleep 10" are enough slow to have "#if slow".
timeless - Nov. 29, 2015, 6:49 a.m.
Yuya Nishihara wrote:
>> @@ -922,7 +934,12 @@
This is in     def _hghave(self, reqs):

>>              print(stdout)
>>              sys.exit(1)
>>
>> -        return ret == 0
>> +        if ret != 0:
>> +            return False
>> +
>> +        if 'slow' in reqs:
>> +            self._timeout = self._slowtimeout
>
> Perhaps it won't work if there's a "#if slow" conditional.

            elif l.startswith(b'#if'):
                skipping = not self._hghave(lsplit[1:])

It should work :)

> and I think two "sleep 10" are enough slow to have "#if slow".

I wanted my test to show what changes/doesn't change. But I'm flexible.
Yuya Nishihara - Nov. 29, 2015, 8:47 a.m.
On Sun, 29 Nov 2015 01:49:15 -0500, timeless wrote:
> Yuya Nishihara wrote:
> >> @@ -922,7 +934,12 @@
> This is in     def _hghave(self, reqs):
> 
> >>              print(stdout)
> >>              sys.exit(1)
> >>
> >> -        return ret == 0
> >> +        if ret != 0:
> >> +            return False
> >> +
> >> +        if 'slow' in reqs:
> >> +            self._timeout = self._slowtimeout
> >
> > Perhaps it won't work if there's a "#if slow" conditional.
> 
>             elif l.startswith(b'#if'):
>                 skipping = not self._hghave(lsplit[1:])
> 
> It should work :)

Ah, okay, I was wrong. I was afraid that local "#if slow" would affect the
global timeout value, but that is intended. If there is a slow block, the
whole test should be slow.

> > and I think two "sleep 10" are enough slow to have "#if slow".
> 
> I wanted my test to show what changes/doesn't change. But I'm flexible.

Do we need 10sec?
timeless - Nov. 29, 2015, 7:51 p.m.
On Sun, Nov 29, 2015 at 3:47 AM, Yuya Nishihara <yuya@tcha.org> wrote:
> On Sun, 29 Nov 2015 01:49:15 -0500, timeless wrote:
>> Yuya Nishihara wrote:
>> >> @@ -922,7 +934,12 @@
>> This is in     def _hghave(self, reqs):
>>
>> >>              print(stdout)
>> >>              sys.exit(1)
>> >>
>> >> -        return ret == 0
>> >> +        if ret != 0:
>> >> +            return False
>> >> +
>> >> +        if 'slow' in reqs:
>> >> +            self._timeout = self._slowtimeout
>> >
>> > Perhaps it won't work if there's a "#if slow" conditional.
>>
>>             elif l.startswith(b'#if'):
>>                 skipping = not self._hghave(lsplit[1:])
>>
>> It should work :)
>
> Ah, okay, I was wrong. I was afraid that local "#if slow" would affect the
> global timeout value, but that is intended. If there is a slow block, the
> whole test should be slow.
>
>> > and I think two "sleep 10" are enough slow to have "#if slow".
>>
>> I wanted my test to show what changes/doesn't change. But I'm flexible.
>
> Do we need 10sec?

Oh, probably not
sleep 2 and slowtimeout 3 should work fine
Yuya Nishihara - Dec. 1, 2015, 1:03 p.m.
On Sun, 29 Nov 2015 14:51:50 -0500, timeless wrote:
> On Sun, Nov 29, 2015 at 3:47 AM, Yuya Nishihara <yuya@tcha.org> wrote:
> > On Sun, 29 Nov 2015 01:49:15 -0500, timeless wrote:
> >> Yuya Nishihara wrote:
> >> >> @@ -922,7 +934,12 @@
> >> This is in     def _hghave(self, reqs):
> >>
> >> >>              print(stdout)
> >> >>              sys.exit(1)
> >> >>
> >> >> -        return ret == 0
> >> >> +        if ret != 0:
> >> >> +            return False
> >> >> +
> >> >> +        if 'slow' in reqs:
> >> >> +            self._timeout = self._slowtimeout
> >> >
> >> > Perhaps it won't work if there's a "#if slow" conditional.
> >>
> >>             elif l.startswith(b'#if'):
> >>                 skipping = not self._hghave(lsplit[1:])
> >>
> >> It should work :)
> >
> > Ah, okay, I was wrong. I was afraid that local "#if slow" would affect the
> > global timeout value, but that is intended. If there is a slow block, the
> > whole test should be slow.
> >
> >> > and I think two "sleep 10" are enough slow to have "#if slow".
> >>
> >> I wanted my test to show what changes/doesn't change. But I'm flexible.
> >
> > Do we need 10sec?
> 
> Oh, probably not
> sleep 2 and slowtimeout 3 should work fine

Changed timeout values and pushed to the clowncopter, thanks.

Patch

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -154,6 +154,7 @@ 
 defaults = {
     'jobs': ('HGTEST_JOBS', 1),
     'timeout': ('HGTEST_TIMEOUT', 180),
+    'slowtimeout': ('HGTEST_SLOWTIMEOUT', 500),
     'port': ('HGTEST_PORT', 20059),
     'shell': ('HGTEST_SHELL', 'sh'),
 }
@@ -236,6 +237,9 @@ 
     parser.add_option("-t", "--timeout", type="int",
         help="kill errant tests after TIMEOUT seconds"
              " (default: $%s or %d)" % defaults['timeout'])
+    parser.add_option("--slowtimeout", type="int",
+        help="kill errant slow tests after SLOWTIMEOUT seconds"
+             " (default: $%s or %d)" % defaults['slowtimeout'])
     parser.add_option("--time", action="store_true",
         help="time how long each test takes")
     parser.add_option("--json", action="store_true",
@@ -327,7 +331,11 @@ 
         if options.timeout != defaults['timeout']:
             sys.stderr.write(
                 'warning: --timeout option ignored with --debug\n')
+        if options.slowtimeout != defaults['slowtimeout']:
+            sys.stderr.write(
+                'warning: --slowtimeout option ignored with --debug\n')
         options.timeout = 0
+        options.slowtimeout = 0
     if options.py3k_warnings:
         if PYTHON3:
             parser.error(
@@ -430,7 +438,8 @@ 
                  debug=False,
                  timeout=defaults['timeout'],
                  startport=defaults['port'], extraconfigopts=None,
-                 py3kwarnings=False, shell=None):
+                 py3kwarnings=False, shell=None,
+                 slowtimeout=defaults['slowtimeout']):
         """Create a test from parameters.
 
         path is the full path to the file defining the test.
@@ -444,7 +453,9 @@ 
         output.
 
         timeout controls the maximum run time of the test. It is ignored when
-        debug is True.
+        debug is True. See slowtimeout for tests with #require slow.
+
+        slowtimeout overrides timeout if the test has #require slow.
 
         startport controls the starting port number to use for this test. Each
         test will reserve 3 port numbers for execution. It is the caller's
@@ -469,6 +480,7 @@ 
         self._keeptmpdir = keeptmpdir
         self._debug = debug
         self._timeout = timeout
+        self._slowtimeout = slowtimeout
         self._startport = startport
         self._extraconfigopts = extraconfigopts or []
         self._py3kwarnings = py3kwarnings
@@ -922,7 +934,12 @@ 
             print(stdout)
             sys.exit(1)
 
-        return ret == 0
+        if ret != 0:
+            return False
+
+        if 'slow' in reqs:
+            self._timeout = self._slowtimeout
+        return True
 
     def _parsetest(self, lines):
         # We generate a shell script which outputs unique markers to line
diff --git a/tests/test-run-tests.t b/tests/test-run-tests.t
--- a/tests/test-run-tests.t
+++ b/tests/test-run-tests.t
@@ -401,6 +401,34 @@ 
   .
   # Ran 1 tests, 0 skipped, 0 warned, 0 failed.
 
+timeouts
+========
+  $ cat > test-timeout.t <<EOF
+  >   $ sleep 10
+  >   $ echo pass
+  >   pass
+  > EOF
+  > echo '#require slow' > test-slow-timeout.t
+  > cat test-timeout.t >> test-slow-timeout.t
+  $ run-tests.py --with-hg=`which hg` --timeout=1 --slowtimeout=20 test-timeout.t test-slow-timeout.t
+  s
+  ERROR: test-timeout.t output changed
+  !
+  Skipped test-slow-timeout.t: skipped
+  Failed test-timeout.t: timed out
+  # Ran 1 tests, 1 skipped, 0 warned, 1 failed.
+  python hash seed: * (glob)
+  [1]
+  $ run-tests.py --with-hg=`which hg` --timeout=1 --slowtimeout=20 test-timeout.t test-slow-timeout.t --allow-slow-tests
+  .
+  ERROR: test-timeout.t output changed
+  !
+  Failed test-timeout.t: timed out
+  # Ran 2 tests, 0 skipped, 0 warned, 1 failed.
+  python hash seed: * (glob)
+  [1]
+  $ rm test-timeout.t test-slow-timeout.t
+
 test for --time
 ==================