Patchwork run-tests: report path for tests outside current directory

login
register
mail settings
Submitter timeless@mozdev.org
Date Feb. 25, 2016, 8:21 p.m.
Message ID <73b8b9bfb731e673ff05.1456431665@waste.org>
Download mbox | patch
Permalink /patch/13399/
State Changes Requested
Delegated to: Martin von Zweigbergk
Headers show

Comments

timeless@mozdev.org - Feb. 25, 2016, 8:21 p.m.
# HG changeset patch
# User timeless <timeless@mozdev.org>
# Date 1456431250 0
#      Thu Feb 25 20:14:10 2016 +0000
# Node ID 73b8b9bfb731e673ff0501237362701e58dc7016
# Parent  41dcd754526612c43b9695df8851557c851828ef
run-tests: report path for tests outside current directory

Followup for 2836a43c7722
timeless - Feb. 25, 2016, 8:27 p.m.
On Thu, Feb 25, 2016 at 3:21 PM, timeless <timeless@mozdev.org> wrote:
>
> @@ -1640,6 +1645,7 @@
>  # newest.
>
>  def loadtimes(testdir):
> +    # This should only deal with times for tests in testdir
>      times = []
>      try:
>          with open(os.path.join(testdir, '.testtimes-')) as fp:
> @@ -1652,6 +1658,7 @@
>      return times
>
>  def savetimes(testdir, result):
> +    # This should only deal with times for tests in testdir
>      saved = dict(loadtimes(testdir))
>      maxruns = 5
>      skipped = set([str(t[0]) for t in result.skipped])

These hunks are a todo which shouldn't really be included in this commit.

Basically if you have:
test-foo.t
bar/test-foo.t

Then timings for the latter will stomp on the former, which is almost
certainly not wanted.
There's a similar problem w/ xunit reporting.
Pierre-Yves David - Feb. 26, 2016, 9:39 p.m.
On 02/25/2016 09:27 PM, timeless wrote:
> On Thu, Feb 25, 2016 at 3:21 PM, timeless <timeless@mozdev.org> wrote:
>>
>> @@ -1640,6 +1645,7 @@
>>   # newest.
>>
>>   def loadtimes(testdir):
>> +    # This should only deal with times for tests in testdir
>>       times = []
>>       try:
>>           with open(os.path.join(testdir, '.testtimes-')) as fp:
>> @@ -1652,6 +1658,7 @@
>>       return times
>>
>>   def savetimes(testdir, result):
>> +    # This should only deal with times for tests in testdir
>>       saved = dict(loadtimes(testdir))
>>       maxruns = 5
>>       skipped = set([str(t[0]) for t in result.skipped])
>
> These hunks are a todo which shouldn't really be included in this commit.

Can you send a V2 series with them excluded (and the todo in other 
commit(s) if applicable)

Patch

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -450,7 +450,11 @@ 
     # Status code reserved for skipped tests (used by hghave).
     SKIPPED_STATUS = 80
 
-    def __init__(self, path, tmpdir, keeptmpdir=False,
+    def __init__(self,
+                 path,
+                 relpath,
+                 tmpdir,
+                 keeptmpdir=False,
                  debug=False,
                  timeout=defaults['timeout'],
                  startport=defaults['port'], extraconfigopts=None,
@@ -487,6 +491,7 @@ 
         shell is the shell to execute tests in.
         """
         self.path = path
+        self.relpath = relpath
         self.bname = os.path.basename(path)
         self.name = _strpath(self.bname)
         self._testdir = os.path.dirname(path)
@@ -1640,6 +1645,7 @@ 
 # newest.
 
 def loadtimes(testdir):
+    # This should only deal with times for tests in testdir
     times = []
     try:
         with open(os.path.join(testdir, '.testtimes-')) as fp:
@@ -1652,6 +1658,7 @@ 
     return times
 
 def savetimes(testdir, result):
+    # This should only deal with times for tests in testdir
     saved = dict(loadtimes(testdir))
     maxruns = 5
     skipped = set([str(t[0]) for t in result.skipped])
@@ -1701,13 +1708,13 @@ 
 
             if not self._runner.options.noskips:
                 for test, msg in result.skipped:
-                    self.stream.writeln('Skipped %s: %s' % (test.name, msg))
+                    self.stream.writeln('Skipped %s: %s' % (test.relpath, msg))
             for test, msg in result.warned:
-                self.stream.writeln('Warned %s: %s' % (test.name, msg))
+                self.stream.writeln('Warned %s: %s' % (test.relpath, msg))
             for test, msg in result.failures:
-                self.stream.writeln('Failed %s: %s' % (test.name, msg))
+                self.stream.writeln('Failed %s: %s' % (test.relpath, msg))
             for test, msg in result.errors:
-                self.stream.writeln('Errored %s: %s' % (test.name, msg))
+                self.stream.writeln('Errored %s: %s' % (test.relpath, msg))
 
             if self._runner.options.xunit:
                 with open(self._runner.options.xunit, 'wb') as xuf:
@@ -2147,7 +2154,9 @@ 
         refpath = os.path.join(self._testdir, test)
         tmpdir = os.path.join(self._hgtmp, b'child%d' % count)
 
-        t = testcls(refpath, tmpdir,
+        t = testcls(refpath,
+                    test,
+                    tmpdir,
                     keeptmpdir=self.options.keep_tmpdir,
                     debug=self.options.debug,
                     timeout=self.options.timeout,
diff --git a/tests/test-run-tests.t b/tests/test-run-tests.t
--- a/tests/test-run-tests.t
+++ b/tests/test-run-tests.t
@@ -94,8 +94,8 @@ 
   
   ERROR: test-failure-unicode.t output changed
   !
-  Failed test-failure.t: output changed
-  Failed test-failure-unicode.t: output changed
+  Failed $TESTTMP/test-failure.t: output changed
+  Failed $TESTTMP/test-failure-unicode.t: output changed
   # Ran 3 tests, 0 skipped, 0 warned, 2 failed.
   python hash seed: * (glob)
   [1]
@@ -123,8 +123,8 @@ 
   
   ERROR: test-failure-unicode.t output changed
   !
-  Failed test-failure.t: output changed
-  Failed test-failure-unicode.t: output changed
+  Failed $TESTTMP/test-failure.t: output changed
+  Failed $TESTTMP/test-failure-unicode.t: output changed
   # Ran 3 tests, 0 skipped, 0 warned, 2 failed.
   python hash seed: * (glob)
   [1]
@@ -170,7 +170,7 @@ 
   
   ERROR: test-failure.t output changed
   !
-  Failed test-failure.t: output changed
+  Failed $TESTTMP/test-failure.t: output changed
   # Ran 2 tests, 1 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
@@ -204,7 +204,7 @@ 
   
   ERROR: test-failure.t output changed
   !
-  Failed test-failure.t: output changed
+  Failed $TESTTMP/test-failure.t: output changed
   # Ran 1 tests, 0 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
@@ -223,7 +223,7 @@ 
   
   ERROR: test-failure.t output changed
   !
-  Failed test-failure.t: output changed
+  Failed $TESTTMP/test-failure.t: output changed
   # Ran 2 tests, 1 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
@@ -238,7 +238,7 @@ 
   
   ERROR: test-serve-fail.t output changed
   !
-  Failed test-serve-fail.t: server failed to start (HGPORT=*) (glob)
+  Failed $TESTTMP/test-serve-fail.t: server failed to start (HGPORT=*) (glob)
   # Ran 1 tests, 0 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
@@ -289,8 +289,8 @@ 
 
   $ rt --jobs 2 test-failure*.t -n
   !!
-  Failed test-failure*.t: output changed (glob)
-  Failed test-failure*.t: output changed (glob)
+  Failed $TESTTMP/test-failure*.t: output changed (glob)
+  Failed $TESTTMP/test-failure*.t: output changed (glob)
   # Ran 2 tests, 0 skipped, 0 warned, 2 failed.
   python hash seed: * (glob)
   [1]
@@ -310,8 +310,8 @@ 
    This is a noop statement so that
    this test is still more bytes than success.
   
-  Failed test-failure*.t: output changed (glob)
-  Failed test-nothing.t: output changed
+  Failed $TESTTMP/test-failure*.t: output changed (glob)
+  Failed $TESTTMP/test-nothing.t: output changed
   # Ran 2 tests, 0 skipped, 0 warned, 2 failed.
   python hash seed: * (glob)
   [1]
@@ -342,7 +342,7 @@ 
   Accept this change? [n] 
   ERROR: test-failure.t output changed
   !.
-  Failed test-failure.t: output changed
+  Failed $TESTTMP/test-failure.t: output changed
   # Ran 2 tests, 0 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
@@ -360,7 +360,7 @@ 
   Accept this change? [n]* (glob)
   ERROR: test-failure.t output changed
   !.
-  Failed test-failure.t: output changed
+  Failed $TESTTMP/test-failure.t: output changed
   # Ran 2 tests, 0 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
@@ -372,7 +372,7 @@ 
   
   ERROR: test-failure.t output changed
   !.
-  Failed test-failure.t: output changed
+  Failed $TESTTMP/test-failure.t: output changed
   # Ran 2 tests, 0 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
@@ -424,7 +424,7 @@ 
 
   $ rt --nodiff
   !.
-  Failed test-failure.t: output changed
+  Failed $TESTTMP/test-failure.t: output changed
   # Ran 2 tests, 0 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
@@ -448,15 +448,15 @@ 
   > cat test-timeout.t >> test-slow-timeout.t
   $ rt --timeout=1 --slowtimeout=3 test-timeout.t test-slow-timeout.t
   st
-  Skipped test-slow-timeout.t: missing feature: allow slow tests
-  Failed test-timeout.t: timed out
+  Skipped $TESTTMP/test-slow-timeout.t: missing feature: allow slow tests
+  Failed $TESTTMP/test-timeout.t: timed out
   # Ran 1 tests, 1 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
   $ rt --timeout=1 --slowtimeout=3 \
   > test-timeout.t test-slow-timeout.t --allow-slow-tests
   .t
-  Failed test-timeout.t: timed out
+  Failed $TESTTMP/test-timeout.t: timed out
   # Ran 2 tests, 0 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
@@ -490,22 +490,22 @@ 
   > EOF
   $ rt --nodiff
   !.s
-  Skipped test-skip.t: missing feature: nail clipper
-  Failed test-failure.t: output changed
+  Skipped $TESTTMP/test-skip.t: missing feature: nail clipper
+  Failed $TESTTMP/test-failure.t: output changed
   # Ran 2 tests, 1 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
 
   $ rt --keyword xyzzy
   .s
-  Skipped test-skip.t: missing feature: nail clipper
+  Skipped $TESTTMP/test-skip.t: missing feature: nail clipper
   # Ran 2 tests, 2 skipped, 0 warned, 0 failed.
 
 Skips with xml
   $ rt --keyword xyzzy \
   >  --xunit=xunit.xml
   .s
-  Skipped test-skip.t: missing feature: nail clipper
+  Skipped $TESTTMP/test-skip.t: missing feature: nail clipper
   # Ran 2 tests, 2 skipped, 0 warned, 0 failed.
   $ cat xunit.xml
   <?xml version="1.0" encoding="utf-8"?>
@@ -546,8 +546,8 @@ 
   
   ERROR: test-failure.t output changed
   !.s
-  Skipped test-skip.t: missing feature: nail clipper
-  Failed test-failure.t: output changed
+  Skipped $TESTTMP/test-skip.t: missing feature: nail clipper
+  Failed $TESTTMP/test-failure.t: output changed
   # Ran 2 tests, 1 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
@@ -597,7 +597,7 @@ 
    This is a noop statement so that
    this test is still more bytes than success.
   Accept this change? [n] ..s
-  Skipped test-skip.t: missing feature: nail clipper
+  Skipped $TESTTMP/test-skip.t: missing feature: nail clipper
   # Ran 2 tests, 1 skipped, 0 warned, 0 failed.
 
   $ cat report.json
@@ -713,7 +713,7 @@ 
   > EOF
   $ rt $HGTEST_RUN_TESTS_PURE test-very-slow-test.t
   s
-  Skipped test-very-slow-test.t: missing feature: allow slow tests
+  Skipped $TESTTMP/anothertests/test-very-slow-test.t: missing feature: allow slow tests
   # Ran 0 tests, 1 skipped, 0 warned, 0 failed.
   $ rt $HGTEST_RUN_TESTS_PURE --allow-slow-tests test-very-slow-test.t
   .