Patchwork D6452: tests: make run-tests exit non-zero if there are "errors"

login
register
mail settings
Submitter phabricator
Date May 29, 2019, 6:26 a.m.
Message ID <differential-rev-PHID-DREV-c6euqo37jne5sxxucpae-req@phab.mercurial-scm.org>
Download mbox | patch
Permalink /patch/40256/
State Superseded
Headers show

Comments

phabricator - May 29, 2019, 6:26 a.m.
spectral created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Previously, if there was an error such as a broken .t file that caused
  run-tests.py to encounter an exception during parsing, the test would be
  considered in an "errored" state, which is separate from "failed".
  
  The check for whether to exit non-zero or not was based entirely on whether
  there were any tests in a "failed" state, so if there was only an error,
  run-tests would exit with 0. Our test infrastructure would then consider the
  test as passing, causing us to have some tests with false negatives that have
  gone undetected for a few weeks now.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6452

AFFECTED FILES
  tests/run-tests.py

CHANGE DETAILS




To: spectral, #hg-reviewers
Cc: mercurial-devel

Patch

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -2912,7 +2912,7 @@ 
 
                 result = runner.run(suite)
 
-            if result.failures:
+            if result.failures or result.errors:
                 failed = True
 
             result.onEnd()