Patchwork [3,of,3] run-tests: fix test result counts with --keyword specified or skips occurring

login
register
mail settings
Submitter Augie Fackler
Date Aug. 5, 2014, 3:22 p.m.
Message ID <7c725333cd5643556bdf.1407252153@arthedain.pit.corp.google.com>
Download mbox | patch
Permalink /patch/5267/
State Accepted
Headers show

Comments

Augie Fackler - Aug. 5, 2014, 3:22 p.m.
# HG changeset patch
# User Augie Fackler <raf@durin42.com>
# Date 1406591339 14400
#      Mon Jul 28 19:48:59 2014 -0400
# Node ID 7c725333cd5643556bdfe8bbd4c255d65b7e1b10
# Parent  90e315e934033502c91853ab6539f2c30dbf93d3
run-tests: fix test result counts with --keyword specified or skips occurring

This preserves the current behavior that excludes ignored or skipped
tests from the number of tests run, except when tests are ignored due
to the --retest flag.
Matt Mackall - Aug. 5, 2014, 5:24 p.m.
On Tue, 2014-08-05 at 11:22 -0400, Augie Fackler wrote:
> # HG changeset patch
> # User Augie Fackler <raf@durin42.com>
> # Date 1406591339 14400
> #      Mon Jul 28 19:48:59 2014 -0400
> # Node ID 7c725333cd5643556bdfe8bbd4c255d65b7e1b10
> # Parent  90e315e934033502c91853ab6539f2c30dbf93d3
> run-tests: fix test result counts with --keyword specified or skips occurring

Queued for default, thanks.

Patch

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -460,8 +460,15 @@ 
                 raise
             except SkipTest, e:
                 result.addSkip(self, str(e))
+                # The base class will have already counted this as a
+                # test we "ran", but we want to exclude skipped tests
+                # from those we count towards those run.
+                result.testsRun -= 1
             except IgnoreTest, e:
                 result.addIgnore(self, str(e))
+                # As with skips, ignores also should be excluded from
+                # the number of tests executed.
+                result.testsRun -= 1
             except WarnTest, e:
                 result.addWarn(self, str(e))
             except self.failureException, e:
@@ -1101,7 +1108,6 @@ 
     # Polyfill.
     def addSkip(self, test, reason):
         self.skipped.append((test, reason))
-
         if self.showAll:
             self.stream.writeln('skipped %s' % reason)
         else:
@@ -1110,12 +1116,13 @@ 
 
     def addIgnore(self, test, reason):
         self.ignored.append((test, reason))
-
         if self.showAll:
             self.stream.writeln('ignored %s' % reason)
         else:
             if reason != 'not retesting':
                 self.stream.write('i')
+            else:
+                self.testsRun += 1
             self.stream.flush()
 
     def addWarn(self, test, reason):
@@ -1339,18 +1346,8 @@ 
 
         self._runner._checkhglib('Tested')
 
-        # When '--retest' is enabled, only failure tests run. At this point
-        # "result.testsRun" holds the count of failure test that has run. But
-        # as while printing output, we have subtracted the skipped and ignored
-        # count from "result.testsRun". Therefore, to make the count remain
-        # the same, we need to add skipped and ignored count in here.
-        if self._runner.options.retest:
-            result.testsRun = result.testsRun + skipped + ignored
-
-        # This differs from unittest's default output in that we don't count
-        # skipped and ignored tests as part of the total test count.
         self.stream.writeln('# Ran %d tests, %d skipped, %d warned, %d failed.'
-            % (result.testsRun - skipped - ignored,
+            % (result.testsRun,
                skipped + ignored, warned, failed))
         if failed:
             self.stream.writeln('python hash seed: %s' %
diff --git a/tests/test-run-tests.t b/tests/test-run-tests.t
--- a/tests/test-run-tests.t
+++ b/tests/test-run-tests.t
@@ -66,7 +66,7 @@ 
   ERROR: test-failure.t output changed
   !
   Failed test-failure.t: output changed
-  # Ran 1 tests, 1 skipped, 0 warned, 1 failed.
+  # Ran 2 tests, 1 skipped, 0 warned, 1 failed.
   python hash seed: * (glob)
   [1]
 
@@ -79,6 +79,11 @@ 
   .
   # Ran 1 tests, 0 skipped, 0 warned, 0 failed.
 
+success w/ keyword
+  $ $TESTDIR/run-tests.py --with-hg=`which hg` -k xyzzy
+  i.
+  # Ran 1 tests, 1 skipped, 0 warned, 0 failed.
+
 failed
 
   $ $TESTDIR/run-tests.py --with-hg=`which hg` test-failure.t
@@ -99,6 +104,25 @@ 
   python hash seed: * (glob)
   [1]
 
+failure w/ keyword
+  $ $TESTDIR/run-tests.py --with-hg=`which hg` -k rataxes
+  i
+  --- $TESTTMP/test-failure.t
+  +++ $TESTTMP/test-failure.t.err
+  @@ -1,4 +1,4 @@
+     $ echo babar
+  -  rataxes
+  +  babar
+   This is a noop statement so that
+   this test is still more bytes than success.
+  
+  ERROR: test-failure.t output changed
+  !
+  Failed test-failure.t: output changed
+  # Ran 1 tests, 1 skipped, 0 warned, 1 failed.
+  python hash seed: * (glob)
+  [1]
+
 Running In Debug Mode
 ======================
 
@@ -262,3 +286,17 @@ 
   python hash seed: * (glob)
   [1]
 
+  $ $TESTDIR/run-tests.py --with-hg=`which hg` --keyword xyzzy
+  i.s
+  Skipped test-skip.t: irrelevant
+  # Ran 1 tests, 2 skipped, 0 warned, 0 failed.
+
+Missing skips or blacklisted skips don't count as executed:
+  $ echo test-failure.t > blacklist
+  $ $TESTDIR/run-tests.py --with-hg=`which hg` --blacklist=blacklist \
+  >   test-failure.t test-bogus.t
+  ss
+  Skipped test-bogus.t: Doesn't exist
+  Skipped test-failure.t: blacklisted
+  # Ran 0 tests, 2 skipped, 0 warned, 0 failed.
+