Patchwork [16,of,19] snapshot: refine candidate snapshot base upward

login
register
mail settings
Submitter Boris Feld
Date Sept. 8, 2018, 10:57 a.m.
Message ID <2ad2e932a5e2bb5bbf2b.1536404230@localhost.localdomain>
Download mbox | patch
Permalink /patch/34439/
State Accepted
Headers show

Comments

Boris Feld - Sept. 8, 2018, 10:57 a.m.
# HG changeset patch
# User Boris Feld <boris.feld@octobus.net>
# Date 1536333456 14400
#      Fri Sep 07 11:17:36 2018 -0400
# Node ID 2ad2e932a5e2bb5bbf2bac7b7d3870ba969c07d6
# Parent  45a29cbc741ec23c1c3c98227c5ca2fcf6f0a3ff
# EXP-Topic sparse-snapshot
# Available At https://bitbucket.org/octobus/mercurial-devel/
#              hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 2ad2e932a5e2
snapshot: refine candidate snapshot base upward

Once we found a suitable snapshot base it is useful to check if it has a
"children" snapshot that would provide a better diff. This is useful when base
not directly related to stored revision are picked. In those case, we "jumped"
to this new chain at an arbitrary point, checking if a higher point is more
appropriate will help to provide better results and increase snapshot reuse.
Yuya Nishihara - Sept. 13, 2018, 1:24 p.m.
On Sat, 08 Sep 2018 12:57:10 +0200, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld <boris.feld@octobus.net>
> # Date 1536333456 14400
> #      Fri Sep 07 11:17:36 2018 -0400
> # Node ID 2ad2e932a5e2bb5bbf2bac7b7d3870ba969c07d6
> # Parent  45a29cbc741ec23c1c3c98227c5ca2fcf6f0a3ff
> # EXP-Topic sparse-snapshot
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #              hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 2ad2e932a5e2
> snapshot: refine candidate snapshot base upward

I got cryptic revlog error around this change.

Unscientific STR:

$ hg clone https://www.mercurial-scm.org/repo/evolve/ \
> -r7b129ce23fad -r2c60ad0d54a9 --config format.usegeneraldelta=0
destination directory: evolve
adding changesets
adding manifests
adding file changes
added 3872 changesets with 7552 changes to 419 files (+1 heads)
new changesets bbeef801409c:2c60ad0d54a9
updating to branch default
272 files updated, 0 files merged, 0 files removed, 0 files unresolved

$ cd evolve

$ hg pull --traceback
searching for changes
adding changesets
adding manifests
transaction abort!
rollback completed
Traceback (most recent call last):
  File "mercurial/scmutil.py", line 164, in callcatch
    return func()
  File "mercurial/dispatch.py", line 350, in _runcatchfunc
    return _dispatch(req)
  File "mercurial/dispatch.py", line 987, in _dispatch
    cmdpats, cmdoptions)
  File "mercurial/dispatch.py", line 733, in runcommand
    ret = _runcommand(ui, options, cmd, d)
  File "mercurial/dispatch.py", line 995, in _runcommand
    return cmdfunc()
  File "mercurial/dispatch.py", line 984, in <lambda>
    d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
  File "mercurial/util.py", line 1532, in check
    return func(*args, **kwargs)
  File "mercurial/util.py", line 1532, in check
    return func(*args, **kwargs)
  File "hgext/mq.py", line 3602, in mqcommand
    return orig(ui, repo, *args, **kwargs)
  File "mercurial/util.py", line 1532, in check
    return func(*args, **kwargs)
  File "mercurial/util.py", line 1532, in check
    return func(*args, **kwargs)
  File "hgext/rebase.py", line 1843, in pullrebase
    ret = orig(ui, repo, *args, **opts)
  File "mercurial/util.py", line 1532, in check
    return func(*args, **kwargs)
  File "mercurial/commands.py", line 4238, in pull
    opargs=pullopargs).cgresult
  File "mercurial/exchange.py", line 1494, in pull
    _fullpullbundle2(repo, pullop)
  File "mercurial/exchange.py", line 1434, in _fullpullbundle2
    _pullbundle2(pullop)
  File "mercurial/exchange.py", line 1677, in _pullbundle2
    bundle2.processbundle(pullop.repo, bundle, op=op)
  File "mercurial/bundle2.py", line 460, in processbundle
    processparts(repo, op, unbundler)
  File "mercurial/bundle2.py", line 467, in processparts
    _processpart(op, part)
  File "mercurial/bundle2.py", line 534, in _processpart
    handler(op, part)
  File "mercurial/bundle2.py", line 1805, in handlechangegroup
    expectedtotal=nbchangesets, **extrakwargs)
  File "mercurial/bundle2.py", line 470, in _processchangegroup
    ret = cg.apply(op.repo, tr, source, url, **kwargs)
  File "mercurial/changegroup.py", line 329, in apply
    self._unpackmanifests(repo, revmap, trp, progress)
  File "mercurial/changegroup.py", line 256, in _unpackmanifests
    repo.manifestlog.getstorage(b'').addgroup(deltas, revmap, trp)
  File "mercurial/manifest.py", line 1457, in addgroup
    addrevisioncb=addrevisioncb)
  File "mercurial/revlog.py", line 2100, in addgroup
    deltacomputer=deltacomputer)
  File "mercurial/revlog.py", line 1968, in _addrevision
    deltainfo = deltacomputer.finddeltainfo(revinfo, fh)
  File "mercurial/revlogutils/deltas.py", line 888, in finddeltainfo
    candidatedelta = self._builddeltainfo(revinfo, candidaterev, fh)
  File "mercurial/revlogutils/deltas.py", line 806, in _builddeltainfo
    delta = self._builddeltadiff(base, revinfo, fh)
  File "mercurial/revlogutils/deltas.py", line 789, in _builddeltadiff
    t = self.buildtext(revinfo, fh)
  File "mercurial/revlogutils/deltas.py", line 784, in buildtext
    revinfo.flags, revinfo.node)
  File "mercurial/revlogutils/deltas.py", line 454, in _textfromdelta
    basetext = revlog.revision(baserev, _df=fh, raw=False)
  File "mercurial/revlog.py", line 1638, in revision
    self.checkhash(text, node, rev=rev)
  File "mercurial/revlog.py", line 1726, in checkhash
    % (self.indexfile, pycompat.bytestr(revornode)))
RevlogError: integrity check failed on 00manifest.i:2367
abort: integrity check failed on 00manifest.i:2367!

This happens on pull. I got a similar error on update, commit, and shelve,
but I can't reproduce them anymore. In any cases, "hg verify" passed, so
the repository storage should be intact.
Yuya Nishihara - Oct. 22, 2018, 1:03 p.m.
On Thu, 13 Sep 2018 22:24:30 +0900, Yuya Nishihara wrote:
> On Sat, 08 Sep 2018 12:57:10 +0200, Boris Feld wrote:
> > # HG changeset patch
> > # User Boris Feld <boris.feld@octobus.net>
> > # Date 1536333456 14400
> > #      Fri Sep 07 11:17:36 2018 -0400
> > # Node ID 2ad2e932a5e2bb5bbf2bac7b7d3870ba969c07d6
> > # Parent  45a29cbc741ec23c1c3c98227c5ca2fcf6f0a3ff
> > # EXP-Topic sparse-snapshot
> > # Available At https://bitbucket.org/octobus/mercurial-devel/
> > #              hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 2ad2e932a5e2
> > snapshot: refine candidate snapshot base upward
> 
> I got cryptic revlog error around this change.

I got this again. Filed as
https://bz.mercurial-scm.org/show_bug.cgi?id=6006

Patch

diff --git a/mercurial/revlogutils/deltas.py b/mercurial/revlogutils/deltas.py
--- a/mercurial/revlogutils/deltas.py
+++ b/mercurial/revlogutils/deltas.py
@@ -658,6 +658,19 @@  def _refinedgroups(revlog, p1, p2, cache
             if base == nullrev:
                 break
             good = yield (base,)
+        # refine snapshot up
+        #
+        # XXX the _findsnapshots call can be expensive and is "duplicated" with
+        # the one done in `_rawgroups`. Once we start working on performance,
+        # we should make the two logics share this computation.
+        snapshots = collections.defaultdict(list)
+        _findsnapshots(revlog, snapshots, good + 1)
+        previous = None
+        while good != previous:
+            previous = good
+            children = tuple(sorted(c for c in snapshots[good]))
+            good = yield children
+
     # we have found nothing
     yield None
 
diff --git a/tests/test-sparse-revlog.t b/tests/test-sparse-revlog.t
--- a/tests/test-sparse-revlog.t
+++ b/tests/test-sparse-revlog.t
@@ -77,7 +77,7 @@  repeatedly while some of it changes rare
   
 
   $ f -s .hg/store/data/*.d
-  .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=59303048
+  .hg/store/data/_s_p_a_r_s_e-_r_e_v_l_o_g-_t_e_s_t-_f_i_l_e.d: size=59302280
   $ hg debugrevlog *
   format : 1
   flags  : generaldelta
@@ -89,45 +89,45 @@  repeatedly while some of it changes rare
       empty     :        0 ( 0.00%)
                      text  :        0 (100.00%)
                      delta :        0 (100.00%)
-      snapshot  :      165 ( 3.30%)
+      snapshot  :      168 ( 3.36%)
         lvl-0   :              4 ( 0.08%)
-        lvl-1   :             17 ( 0.34%)
-        lvl-2   :             46 ( 0.92%)
-        lvl-3   :             62 ( 1.24%)
-        lvl-4   :             36 ( 0.72%)
-      deltas    :     4836 (96.70%)
-  revision size : 59303048
-      snapshot  :  6105443 (10.30%)
-        lvl-0   :         804187 ( 1.36%)
-        lvl-1   :        1476228 ( 2.49%)
-        lvl-2   :        1752567 ( 2.96%)
-        lvl-3   :        1461776 ( 2.46%)
-        lvl-4   :         610685 ( 1.03%)
-      deltas    : 53197605 (89.70%)
+        lvl-1   :             18 ( 0.36%)
+        lvl-2   :             39 ( 0.78%)
+        lvl-3   :             54 ( 1.08%)
+        lvl-4   :             53 ( 1.06%)
+      deltas    :     4833 (96.64%)
+  revision size : 59302280
+      snapshot  :  5833942 ( 9.84%)
+        lvl-0   :         804068 ( 1.36%)
+        lvl-1   :        1378470 ( 2.32%)
+        lvl-2   :        1608138 ( 2.71%)
+        lvl-3   :        1222158 ( 2.06%)
+        lvl-4   :         821108 ( 1.38%)
+      deltas    : 53468338 (90.16%)
   
   chunks        :     5001
       0x78 (x)  :     5001 (100.00%)
-  chunks size   : 59303048
-      0x78 (x)  : 59303048 (100.00%)
+  chunks size   : 59302280
+      0x78 (x)  : 59302280 (100.00%)
   
   avg chain length  :       17
   max chain length  :       45
-  max chain reach   : 26194433
+  max chain reach   : 22744720
   compression ratio :       29
   
   uncompressed data size (min/max/avg) : 346468 / 346472 / 346471
-  full revision size (min/max/avg)     : 200992 / 201080 / 201046
-  inter-snapshot size (min/max/avg)    : 11610 / 172762 / 32927
-      level-1   (min/max/avg)          : 15619 / 172762 / 86836
-      level-2   (min/max/avg)          : 13055 / 85219 / 38099
-      level-3   (min/max/avg)          : 11610 / 42645 / 23577
-      level-4   (min/max/avg)          : 12928 / 20205 / 16963
-  delta size (min/max/avg)             : 10649 / 106863 / 11000
+  full revision size (min/max/avg)     : 200985 / 201050 / 201017
+  inter-snapshot size (min/max/avg)    : 11598 / 163304 / 30669
+      level-1   (min/max/avg)          : 15616 / 163304 / 76581
+      level-2   (min/max/avg)          : 11602 / 86428 / 41234
+      level-3   (min/max/avg)          : 11598 / 42390 / 22632
+      level-4   (min/max/avg)          : 11603 / 19649 / 15492
+  delta size (min/max/avg)             : 10649 / 105465 / 11063
   
-  deltas against prev  : 4162 (86.06%)
-      where prev = p1  : 4120     (98.99%)
+  deltas against prev  : 4167 (86.22%)
+      where prev = p1  : 4129     (99.09%)
       where prev = p2  :    0     ( 0.00%)
-      other            :   42     ( 1.01%)
-  deltas against p1    :  653 (13.50%)
-  deltas against p2    :   21 ( 0.43%)
+      other            :   38     ( 0.91%)
+  deltas against p1    :  643 (13.30%)
+  deltas against p2    :   23 ( 0.48%)
   deltas against other :    0 ( 0.00%)