Patchwork [cvs-tests-yet-again] cvsps: use a different tiebreaker to avoid flaky test

login
register
mail settings
Submitter Augie Fackler
Date March 13, 2015, 7:11 p.m.
Message ID <7379dd15939f06e3b8fc.1426273891@arthedain.pit.corp.google.com>
Download mbox | patch
Permalink /patch/8059/
State Accepted
Commit 867c3649be5d82f1eee2be6895aa26e5d5e53433
Headers show

Comments

Augie Fackler - March 13, 2015, 7:11 p.m.
# HG changeset patch
# User Augie Fackler <raf@durin42.com>
# Date 1426270813 14400
#      Fri Mar 13 14:20:13 2015 -0400
# Node ID 7379dd15939f06e3b8fcea39132e614ebf8dd6f2
# Parent  2b7ab29627fd93ca7f5cb838403c2f6c728469bd
cvsps: use a different tiebreaker to avoid flaky test

After adding some sneaky debug printing[0], I determined that this
test flaked when a CVS commit containing two files starts too close to
the end of a second, thus putting file "a" in one second and "b/c" in
the following second. The secondary sort key meant that these changes
sorted in a different order when the timestamps were different than
they did when they matched. As far as I can tell, CVS walks through
the files in a stable order, so by sorting on the filenames in cvsps
we'll get stable output. It's fine for us to switch from sorting on
the branchpoint as a secondary key because this was already the point
when we didn't care, and we're just trying to break ties in a stable
way. It's unclear to be if having the branchpoint present matters
anymore, but it doesn't really hurt to leave it.

With this change in place, I was able to run test-convert-cvs over 650
times in a row without a failure.  test-convert-cvcs-synthetic.t
appears to still be flaky, but I don't think it's *worse* than it was
before - just not better (I observed one flaky failure in 200 runs on
that test).

0: The helpful debug hack ended up being this, in case it's useful to
future flaky test assassins:

 --- a/hgext/convert/cvsps.py
 +++ b/hgext/convert/cvsps.py
 @@ -854,6 +854,8 @@ def debugcvsps(ui, *args, **opts):
              ui.write(('Branch: %s\n' % (cs.branch or 'HEAD')))
              ui.write(('Tag%s: %s \n' % (['', 's'][len(cs.tags) > 1],
                                    ','.join(cs.tags) or '(none)')))
 +            if cs.comment == 'ci1' and (cs.id == 6) == bool(cs.branchpoints):
 +                ui.write('raw timestamp %r\n' % (cs.date,))
              if cs.branchpoints:
                  ui.write(('Branchpoints: %s \n') %
                           ', '.join(sorted(cs.branchpoints)))
Matt Mackall - March 14, 2015, 9:58 p.m.
On Fri, 2015-03-13 at 15:11 -0400, Augie Fackler wrote:
> # HG changeset patch
> # User Augie Fackler <raf@durin42.com>
> # Date 1426270813 14400
> #      Fri Mar 13 14:20:13 2015 -0400
> # Node ID 7379dd15939f06e3b8fcea39132e614ebf8dd6f2
> # Parent  2b7ab29627fd93ca7f5cb838403c2f6c728469bd
> cvsps: use a different tiebreaker to avoid flaky test

Queued for default, thanks.

Patch

diff --git a/hgext/convert/cvsps.py b/hgext/convert/cvsps.py
--- a/hgext/convert/cvsps.py
+++ b/hgext/convert/cvsps.py
@@ -634,14 +634,21 @@  def createchangeset(ui, log, fuzz=60, me
         # By this point, the changesets are sufficiently compared that
         # we don't really care about ordering. However, this leaves
         # some race conditions in the tests, so we compare on the
-        # number of files modified and the number of branchpoints in
-        # each changeset to ensure test output remains stable.
+        # number of files modified, the files contained in each
+        # changeset, and the branchpoints in the change to ensure test
+        # output remains stable.
 
         # recommended replacement for cmp from
         # https://docs.python.org/3.0/whatsnew/3.0.html
         c = lambda x, y: (x > y) - (x < y)
+        # Sort bigger changes first.
         if not d:
             d = c(len(l.entries), len(r.entries))
+        # Try sorting by filename in the change.
+        if not d:
+            d = c([e.file for e in l.entries], [e.file for e in r.entries])
+        # Try and put changes without a branch point before ones with
+        # a branch point.
         if not d:
             d = c(len(l.branchpoints), len(r.branchpoints))
         return d
diff --git a/tests/test-convert-cvs.t b/tests/test-convert-cvs.t
--- a/tests/test-convert-cvs.t
+++ b/tests/test-convert-cvs.t
@@ -397,11 +397,12 @@  testing debugcvsps
   Author: * (glob)
   Branch: HEAD
   Tag: (none) 
+  Branchpoints: branch 
   Log:
   ci1
   
   Members: 
-  	b/c:1.2->1.3 
+  	a:1.1->1.2 
   
   ---------------------
   PatchSet 6 
@@ -409,12 +410,11 @@  testing debugcvsps
   Author: * (glob)
   Branch: HEAD
   Tag: (none) 
-  Branchpoints: branch 
   Log:
   ci1
   
   Members: 
-  	a:1.1->1.2 
+  	b/c:1.2->1.3 
   
   ---------------------
   PatchSet 7