Patchwork [1,of,6] run-tests: wait for test threads after first error

login
register
mail settings
Submitter Gregory Szorc
Date March 29, 2015, 3:15 a.m.
Message ID <3c820c7b3ab9657c8b2d.1427598906@vm-ubuntu-main.gateway.sonic.net>
Download mbox | patch
Permalink /patch/8336/
State Accepted
Headers show

Comments

Gregory Szorc - March 29, 2015, 3:15 a.m.
# HG changeset patch
# User Gregory Szorc <gregory.szorc@gmail.com>
# Date 1427596743 25200
#      Sat Mar 28 19:39:03 2015 -0700
# Node ID 3c820c7b3ab9657c8b2d0a769ad2385c09371eac
# Parent  bebbe08335400096292c194c2a860cc4318ff648
run-tests: wait for test threads after first error

The test runner has the ability to stop on first error.

Tests are executed in new Python threads. The test runner starts new
threads when it has capacity to do so. Before this patch, the "stop on
first error" logic would return immediately from the "run tests"
function, without waiting on test threads to complete. There was thus
a race between the test runner thread doing cleanup work and the test
thread performing activity. For example, the test thread could be in
the middle of executing a test shell script and the test runner
could remove the test's temporary directory. Depending on timing, this
could result in any number of output from the test runner.

This patch eliminates the race condition by having the test runner
explicitly wait for test threads to complete before continuing.

I discovered this issue as I modified the test harness in a subsequent
patch and was reliably able to tickle the race condition.

Patch

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -1389,18 +1389,21 @@  class TestSuite(unittest.TestSuite):
             except: # re-raises
                 done.put(('!', test, 'run-test raised an error, see traceback'))
                 raise
 
+        stoppedearly = False
+
         try:
             while tests or running:
                 if not done.empty() or running == self._jobs or not tests:
                     try:
                         done.get(True, 1)
+                        running -= 1
                         if result and result.shouldStop:
+                            stoppedearly = True
                             break
                     except queue.Empty:
                         continue
-                    running -= 1
                 if tests and not running == self._jobs:
                     test = tests.pop(0)
                     if self._loop:
                         if getattr(test, 'should_reload', False):
@@ -1412,8 +1415,20 @@  class TestSuite(unittest.TestSuite):
                     t = threading.Thread(target=job, name=test.name,
                                          args=(test, result))
                     t.start()
                     running += 1
+
+            # If we stop early we still need to wait on started tests to
+            # finish. Otherwise, there is a race between the test completing
+            # and the test's cleanup code running. This could result in the
+            # test reporting incorrect.
+            if stoppedearly:
+                while running:
+                    try:
+                        done.get(True, 1)
+                        running -= 1
+                    except queue.Empty:
+                        continue
         except KeyboardInterrupt:
             for test in runtests:
                 test.abort()
 
diff --git a/tests/test-run-tests.t b/tests/test-run-tests.t
--- a/tests/test-run-tests.t
+++ b/tests/test-run-tests.t
@@ -229,9 +229,10 @@  failures in parallel with --first should
    This is a noop statement so that
    this test is still more bytes than success.
   
   Failed test-failure*.t: output changed (glob)
-  # Ran 2 tests, 0 skipped, 0 warned, 1 failed.
+  Failed test-nothing.t: output changed
+  # Ran 2 tests, 0 skipped, 0 warned, 2 failed.
   python hash seed: * (glob)
   [1]