Patchwork D10695: cache: avoid warming the fnodetags cache after clone

login
register
mail settings
Submitter phabricator
Date May 7, 2021, 9:33 a.m.
Message ID <differential-rev-PHID-DREV-sfcmd6l4olbaoasezvg2-req@mercurial-scm.org>
Download mbox | patch
Permalink /patch/49014/
State Superseded
Headers show

Comments

phabricator - May 7, 2021, 9:33 a.m.
marmoute created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  That cache can quite expensive to compute on large repository as not that `hg
  clone` is warming all cache, this can introduces a significant slowdown for
  clone time[1]. As a stop gap measure introduce a quick fix for
  that on stable, skipping the fnodetags cache post-clone.
  
  [1] https://www.mercurial-scm.org/pipermail/mercurial/2021-April/052679.html

REPOSITORY
  rHG Mercurial

BRANCH
  stable

REVISION DETAIL
  https://phab.mercurial-scm.org/D10695

AFFECTED FILES
  mercurial/hg.py
  mercurial/localrepo.py
  tests/test-clone-uncompressed.t
  tests/test-clone.t
  tests/test-hardlinks.t
  tests/test-share.t
  tests/test-ssh.t
  tests/test-tags.t
  tests/test-treemanifest.t

CHANGE DETAILS




To: marmoute, #hg-reviewers
Cc: mercurial-patches, mercurial-devel

Patch

diff --git a/tests/test-treemanifest.t b/tests/test-treemanifest.t
--- a/tests/test-treemanifest.t
+++ b/tests/test-treemanifest.t
@@ -792,7 +792,7 @@ 
   $ hg clone --config experimental.changegroup3=True --stream -U \
   >   http://localhost:$HGPORT1 stream-clone-basicstore
   streaming all changes
-  29 files to transfer, * of data (glob)
+  28 files to transfer, * of data (glob)
   transferred * in * seconds (*) (glob)
   $ hg -R stream-clone-basicstore verify
   checking changesets
@@ -806,7 +806,7 @@ 
   $ hg clone --config experimental.changegroup3=True --stream -U \
   >   http://localhost:$HGPORT2 stream-clone-encodedstore
   streaming all changes
-  29 files to transfer, * of data (glob)
+  28 files to transfer, * of data (glob)
   transferred * in * seconds (*) (glob)
   $ hg -R stream-clone-encodedstore verify
   checking changesets
diff --git a/tests/test-tags.t b/tests/test-tags.t
--- a/tests/test-tags.t
+++ b/tests/test-tags.t
@@ -807,11 +807,11 @@ 
 
   $ f --size --hexdump tagsclient/.hg/cache/hgtagsfnodes1
   tagsclient/.hg/cache/hgtagsfnodes1: size=96
-  0000: 96 ee 1d 73 00 00 00 00 00 00 00 00 00 00 00 00 |...s............|
-  0010: 00 00 00 00 00 00 00 00 c4 da b0 c2 94 65 e1 c6 |.............e..|
-  0020: 0d f7 f0 dd 32 04 ea 57 78 c8 97 97 79 fc d5 95 |....2..Wx...y...|
-  0030: f6 3c c8 fe 94 65 e1 c6 0d f7 f0 dd 32 04 ea 57 |.<...e......2..W|
-  0040: 78 c8 97 97 79 fc d5 95 40 f0 35 8c 19 e0 a7 d3 |x...y...@.5.....|
+  0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+  0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+  0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+  0030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+  0040: ff ff ff ff ff ff ff ff 40 f0 35 8c 19 e0 a7 d3 |........@.5.....|
   0050: 8a 5c 6a 82 4d cf fb a5 87 d0 2f a3 1e 4f 2f 8a |.\j.M...../..O/.|
 
 Running hg tags should produce tags2* file and not change cache
@@ -837,11 +837,11 @@ 
 
   $ f --size --hexdump tagsclient/.hg/cache/hgtagsfnodes1
   tagsclient/.hg/cache/hgtagsfnodes1: size=96
-  0000: 96 ee 1d 73 00 00 00 00 00 00 00 00 00 00 00 00 |...s............|
-  0010: 00 00 00 00 00 00 00 00 c4 da b0 c2 94 65 e1 c6 |.............e..|
-  0020: 0d f7 f0 dd 32 04 ea 57 78 c8 97 97 79 fc d5 95 |....2..Wx...y...|
-  0030: f6 3c c8 fe 94 65 e1 c6 0d f7 f0 dd 32 04 ea 57 |.<...e......2..W|
-  0040: 78 c8 97 97 79 fc d5 95 40 f0 35 8c 19 e0 a7 d3 |x...y...@.5.....|
+  0000: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+  0010: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+  0020: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+  0030: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff |................|
+  0040: ff ff ff ff ff ff ff ff 40 f0 35 8c 19 e0 a7 d3 |........@.5.....|
   0050: 8a 5c 6a 82 4d cf fb a5 87 d0 2f a3 1e 4f 2f 8a |.\j.M...../..O/.|
 
 Check that the bundle includes cache data
diff --git a/tests/test-ssh.t b/tests/test-ssh.t
--- a/tests/test-ssh.t
+++ b/tests/test-ssh.t
@@ -86,7 +86,7 @@ 
   $ hg -R local-stream book mybook
   $ hg clone -e "\"$PYTHON\" \"$TESTDIR/dummyssh\"" --stream ssh://user@dummy/local-stream stream2
   streaming all changes
-  16 files to transfer, * of data (glob)
+  15 files to transfer, * of data (glob)
   transferred * in * seconds (*) (glob)
   updating to branch default
   2 files updated, 0 files merged, 0 files removed, 0 files unresolved
diff --git a/tests/test-share.t b/tests/test-share.t
--- a/tests/test-share.t
+++ b/tests/test-share.t
@@ -68,7 +68,6 @@ 
   branch2-served.hidden
   branch2-visible
   branch2-visible-hidden
-  hgtagsfnodes1
   rbc-names-v1
   rbc-revs-v1
   tags2
diff --git a/tests/test-hardlinks.t b/tests/test-hardlinks.t
--- a/tests/test-hardlinks.t
+++ b/tests/test-hardlinks.t
@@ -244,7 +244,6 @@ 
   2 r4/.hg/cache/branch2-served.hidden
   2 r4/.hg/cache/branch2-visible
   2 r4/.hg/cache/branch2-visible-hidden
-  2 r4/.hg/cache/hgtagsfnodes1
   2 r4/.hg/cache/rbc-names-v1
   2 r4/.hg/cache/rbc-revs-v1
   2 r4/.hg/cache/tags2
@@ -302,7 +301,6 @@ 
   2 r4/.hg/cache/branch2-served.hidden
   2 r4/.hg/cache/branch2-visible
   2 r4/.hg/cache/branch2-visible-hidden
-  2 r4/.hg/cache/hgtagsfnodes1
   2 r4/.hg/cache/rbc-names-v1
   2 r4/.hg/cache/rbc-revs-v1
   2 r4/.hg/cache/tags2
diff --git a/tests/test-clone.t b/tests/test-clone.t
--- a/tests/test-clone.t
+++ b/tests/test-clone.t
@@ -62,7 +62,6 @@ 
   branch2-served.hidden
   branch2-visible
   branch2-visible-hidden
-  hgtagsfnodes1
   rbc-names-v1
   rbc-revs-v1
   tags2
@@ -142,7 +141,6 @@ 
   branch2-served.hidden
   branch2-visible
   branch2-visible-hidden
-  hgtagsfnodes1
   rbc-names-v1
   rbc-revs-v1
   tags2
diff --git a/tests/test-clone-uncompressed.t b/tests/test-clone-uncompressed.t
--- a/tests/test-clone-uncompressed.t
+++ b/tests/test-clone-uncompressed.t
@@ -201,7 +201,6 @@ 
   branch2-served.hidden
   branch2-visible
   branch2-visible-hidden
-  hgtagsfnodes1
   rbc-names-v1
   rbc-revs-v1
   tags2
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -2727,6 +2727,11 @@ 
 
         If 'full' is set, make sure all caches the function knows about have
         up-to-date data. Even the ones usually loaded more lazily.
+
+        The `full` argument can take a special "post-clone" value. In this case
+        the cache warming is made after a clone and of the slower cache might
+        be skipped, namely the `.fnodetags` one. This argument is 5.8 specific
+        as we plan for a cleaner way to deal with this for 5.9.
         """
         if tr is not None and tr.hookargs.get(b'source') == b'strip':
             # During strip, many caches are invalid but
@@ -2754,8 +2759,9 @@ 
             for ctx in self[b'.'].parents():
                 ctx.manifest()  # accessing the manifest is enough
 
-            # accessing fnode cache warms the cache
-            tagsmod.fnoderevs(self.ui, unfi, unfi.changelog.revs())
+            if not full == b"post-clone":
+                # accessing fnode cache warms the cache
+                tagsmod.fnoderevs(self.ui, unfi, unfi.changelog.revs())
             # accessing tags warm the cache
             self.tags()
             self.filtered(b'served').tags()
diff --git a/mercurial/hg.py b/mercurial/hg.py
--- a/mercurial/hg.py
+++ b/mercurial/hg.py
@@ -1053,7 +1053,7 @@ 
             # as the only "bad" outcome would be some slowness. That potential
             # slowness already affect reader.
             with destrepo.lock():
-                destrepo.updatecaches(full=True)
+                destrepo.updatecaches(full=b"post-clone")
     finally:
         release(srclock, destlock)
         if cleandir is not None: