Patchwork [2,of,4,tags-cache-split] tags: use a different cache file for each repo filter (issue4550)

login
register
mail settings
Submitter Gregory Szorc
Date March 30, 2015, 6:14 a.m.
Message ID <c19ef55be215f68a0955.1427696044@vm-ubuntu-main.gateway.sonic.net>
Download mbox | patch
Permalink /patch/8367/
State Accepted
Headers show

Comments

Gregory Szorc - March 30, 2015, 6:14 a.m.
# HG changeset patch
# User Gregory Szorc <gregory.szorc@gmail.com>
# Date 1427658317 25200
#      Sun Mar 29 12:45:17 2015 -0700
# Node ID c19ef55be215f68a0955c49473c2e07b721998ff
# Parent  31716032c6e7f8764a02b212dac872cc3436c052
tags: use a different cache file for each repo filter (issue4550)

(This patch is based on an RFC patch by Pierre-Yves David. I have
modified it to be compatible with the larger tags cache refactor
series.)

Currently, there is a single tags cache file. Whichever filter is
active when tags are requested will write tags data to disk with that
filter's view. This creates issues when different filters access tags
data. For example, for users of obsolescence, the "visible" and
"unfiltered" filters both update the tags cache. The "unfiltered"
filter will write .hgtags filenodes for hidden heads. When the
"visible" filter writes the tags cache, the hidden heads won't be
included because they are not in that view. The next time the
"unfiltered" filter reads the tags cache, it will have to re-compute
.hgtags filenodes for hidden heads. On large repositories (where
.hgtags filenode lookup takes a long time) with many hidden heads,
a significant performance could be encountered as hundreds of even
thousands of .hgtags filenodes take potentially minutes to resolve.
See issue4550 for more details.

This patch introduces per-filter tags cache files. The different
repo filters no longer remove state beloning to each other and
no costly performance penalty is incurred.

The patch intentionally introduces a clean break from the old cache
file. The old .hg/cache/tags file is no longer produced nor read.
This is done so an upcoming patch can change the format of the
file without concerns about older clients accessing a newer cache
and encountering corruption. This isn't strictly necessary, as cache
reading is inside a block that catches Exception. But, introducing
new per-filter files is essentially creating a clean break anyway,
so we might as well make it consistent.

A version number was introduced to the cache file. This is similar to
how other caches work. It provides an alternate vector for future
upgrades.
Matt Mackall - March 31, 2015, 1:04 p.m.
On Sun, 2015-03-29 at 23:14 -0700, Gregory Szorc wrote:
> The patch intentionally introduces a clean break from the old cache
> file. The old .hg/cache/tags file is no longer produced nor read.
> This is done so an upcoming patch can change the format of the
> file without concerns about older clients accessing a newer cache
> and encountering corruption. This isn't strictly necessary, as cache
> reading is inside a block that catches Exception. But, introducing
> new per-filter files is essentially creating a clean break anyway,
> so we might as well make it consistent.

This makes more sense than my first reading of the previous patch. Still
not sold on migrating cache data.

> A version number was introduced to the cache file. This is similar to
> how other caches work. It provides an alternate vector for future
> upgrades.

It is? Oh, I see, a number is added to the name, not the file itself.

> -            ui.warn(_('.hg/cache/tags is corrupt, rebuilding it\n'))
> +            ui.warn(_('.hg/%s is corrupt, rebuilding it\n') % _filename(repo))

While we're here, this message shouldn't exist: caches should be
transparent. Google says no one's ever hit it so I'll just rip it out.

Patch

diff --git a/mercurial/tags.py b/mercurial/tags.py
--- a/mercurial/tags.py
+++ b/mercurial/tags.py
@@ -66,8 +66,12 @@  import time
 #
 # Tags associated with multiple changesets have an entry for each changeset.
 # The most recent changeset (in terms of revlog ordering for the head
 # setting it) for each tag is last.
+#
+# There are separate cache files for each repository filter. This is
+# so each repoview can have its own state without overwriting the state
+# of others. See issue4550 for some background.
 
 def findglobaltags(ui, repo, alltags, tagtypes):
     '''Find global tags in a repo.
 
@@ -246,8 +250,12 @@  def _updatetags(filetags, tagtype, allta
             tagtypes[name] = tagtype
         ahist.extend([n for n in bhist if n not in ahist])
         alltags[name] = anode, ahist
 
+def _filename(repo):
+    """name of a tagcache file for a given repo or repoview"""
+    return 'cache/tags1-%s' % (repo.filtername or 'unfiltered')
+
 def _readtagcache(ui, repo):
     '''Read the tag cache.
 
     Returns a tuple (heads, fnodes, cachetags, shouldwrite).
@@ -263,11 +271,10 @@  def _readtagcache(ui, repo):
 
     If the cache is not up to date, the caller is responsible for reading tag
     info from each returned head. (See findglobaltags().)
     '''
-
     try:
-        cachefile = repo.vfs('cache/tags', 'r')
+        cachefile = repo.vfs(_filename(repo), 'r')
         # force reading the file for static-http
         cachelines = iter(cachefile)
     except IOError:
         cachefile = None
@@ -293,9 +300,9 @@  def _readtagcache(ui, repo):
                     fnode = bin(line[2])
                     cachefnode[headnode] = fnode
         except Exception:
             # corruption of the tags cache, just recompute it
-            ui.warn(_('.hg/cache/tags is corrupt, rebuilding it\n'))
+            ui.warn(_('.hg/%s is corrupt, rebuilding it\n') % _filename(repo))
             cacheheads = []
             cacherevs = []
             cachefnode = {}
 
@@ -312,9 +319,9 @@  def _readtagcache(ui, repo):
         cachefile = None
         # If tag reading had issues, fall through and repair the cache.
         if not warned:
             return (None, None, tags, False)
-        ui.warn(_('.hg/cache/tags is corrupt; rebuilding it\n'))
+        ui.warn(_('.hg/%s is corrupt; rebuilding it\n') % _filename(repo))
     if cachefile:
         cachefile.close()               # ignore rest of file
 
     repoheads = repo.heads()
@@ -371,9 +378,9 @@  def _readtagcache(ui, repo):
     return (repoheads, cachefnode, None, True)
 
 def _writetagcache(ui, repo, heads, tagfnode, cachetags):
     try:
-        cachefile = repo.vfs('cache/tags', 'w', atomictemp=True)
+        cachefile = repo.vfs(_filename(repo), 'w', atomictemp=True)
     except (OSError, IOError):
         return
 
     ui.log('tagscache', 'writing tags cache file with %d heads and %d tags\n',
diff --git a/tests/test-mq.t b/tests/test-mq.t
--- a/tests/test-mq.t
+++ b/tests/test-mq.t
@@ -310,14 +310,14 @@  qpop
 
 qpush with dump of tag cache
 Dump the tag cache to ensure that it has exactly one head after qpush.
 
-  $ rm -f .hg/cache/tags
+  $ rm -f .hg/cache/tags1-visible
   $ hg tags > /dev/null
 
-.hg/cache/tags (pre qpush):
+.hg/cache/tags1-visible (pre qpush):
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags1-visible
   1 [\da-f]{40} (re)
   
   $ hg qpush
   applying test.patch
@@ -325,11 +325,11 @@  Dump the tag cache to ensure that it has
   $ hg phase -r qbase
   2: draft
   $ hg tags > /dev/null
 
-.hg/cache/tags (post qpush):
+.hg/cache/tags1-visible (post qpush):
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags1-visible
   2 [\da-f]{40} (re)
   
   $ checkundo qpush
   $ cd ..
diff --git a/tests/test-obsolete-tag-cache.t b/tests/test-obsolete-tag-cache.t
--- a/tests/test-obsolete-tag-cache.t
+++ b/tests/test-obsolete-tag-cache.t
@@ -35,9 +35,9 @@  Trigger tags cache population by doing s
   |/
   o  0:55482a6fb4b1 test1 initial
   
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags1-visible
   4 042eb6bfcc4909bad84a1cbf6eb1ddf0ab587d41
   3 c3cb30f2d2cd0aae008cc91a07876e3c5131fd22 b3bce87817fe7ac9dca2834366c1d7534c095cf1
   
   55482a6fb4b1881fa8f746fd52cf6f096bb21c89 test1
@@ -58,13 +58,40 @@  repopulation
   |
   o  0:55482a6fb4b1 test1 initial
   
 
-.hgtags filenodes for hidden heads should be visible (issue4550)
-(currently broken)
+Trigger population on unfiltered repo
 
-  $ cat .hg/cache/tags
+  $ hg --hidden log -G -T '{rev}:{node|short} {tags} {desc}\n'
+  o  7:eb610439e10e tip test2 tag
+  |
+  o  6:7b4af00c3c83  first
+  |
+  o  5:43ac2a539b3c  test tag
+  |
+  @  4:042eb6bfcc49  newhead
+  |
+  | x  3:c3cb30f2d2cd  test2 tag
+  | |
+  | x  2:d75775ffbc6b test2 first
+  | |
+  | x  1:5f97d42da03f  test tag
+  |/
+  o  0:55482a6fb4b1 test1 initial
+  
+
+visible cache should only contain visible head (issue4550)
+
+  $ cat .hg/cache/tags1-visible
   7 eb610439e10e0c6b296f97b59624c2e24fc59e30 b3bce87817fe7ac9dca2834366c1d7534c095cf1
   
   55482a6fb4b1881fa8f746fd52cf6f096bb21c89 test1
   d75775ffbc6bca1794d300f5571272879bd280da test2
 
+unfiltered cache should contain all topological heads (issue4550)
+
+  $ cat .hg/cache/tags1-unfiltered
+  7 eb610439e10e0c6b296f97b59624c2e24fc59e30 b3bce87817fe7ac9dca2834366c1d7534c095cf1
+  3 c3cb30f2d2cd0aae008cc91a07876e3c5131fd22 b3bce87817fe7ac9dca2834366c1d7534c095cf1
+  
+  55482a6fb4b1881fa8f746fd52cf6f096bb21c89 test1
+  d75775ffbc6bca1794d300f5571272879bd280da test2
diff --git a/tests/test-tags.t b/tests/test-tags.t
--- a/tests/test-tags.t
+++ b/tests/test-tags.t
@@ -1,8 +1,8 @@ 
 Helper functions:
 
   $ cacheexists() {
-  >   [ -f .hg/cache/tags ] && echo "tag cache exists" || echo "no tag cache"
+  >   [ -f .hg/cache/tags1-visible ] && echo "tag cache exists" || echo "no tag cache"
   > }
 
   $ dumptags() {
   >     rev=$1
@@ -35,11 +35,11 @@  Setup:
   tag cache exists
 
 Try corrupting the cache
 
-  $ printf 'a b' > .hg/cache/tags
+  $ printf 'a b' > .hg/cache/tags1-visible
   $ hg identify
-  .hg/cache/tags is corrupt, rebuilding it
+  .hg/cache/tags1-visible is corrupt, rebuilding it
   acb14030fe0a tip
   $ cacheexists
   tag cache exists
   $ hg identify
@@ -68,16 +68,16 @@  Create a tag behind hg's back:
   b9154636be93 tip
 
 Repeat with cold tag cache:
 
-  $ rm -f .hg/cache/tags
+  $ rm -f .hg/cache/tags1-visible
   $ hg identify
   b9154636be93 tip
 
 And again, but now unable to write tag cache:
 
 #if unix-permissions
-  $ rm -f .hg/cache/tags
+  $ rm -f .hg/cache/tags1-visible
   $ chmod 555 .hg
   $ hg identify
   b9154636be93 tip
   $ chmod 755 .hg
@@ -158,9 +158,9 @@  Report tag parse error on other head:
   .hgtags@c4be69a18c11, line 2: node 'x' is not well formed
   tip                                8:c4be69a18c11
   first                              0:acb14030fe0a
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags1-visible
   8 c4be69a18c11e8bc3a5fdbb576017c25f7d84663 9876b1193cfc564b1518d1f1b4459028ec75bf18
   7 75d9f02dfe2874aa938ee8c18fa27c1328cfb023 7371bc5168f70e1b7c8dbf7c8bedf9d79f51dd82
   
   acb14030fe0a21b60322c440ad2d20cf7685a376 first
@@ -177,19 +177,19 @@  Report tag parse error on other head:
 Test a similar corruption issue in the tags cache
 We should see a warning when loading the tags cache and a repaired version
 should be written automatically.
 
-  $ cat > .hg/cache/tags << EOF
+  $ cat > .hg/cache/tags1-visible << EOF
   > 8 c4be69a18c11e8bc3a5fdbb576017c25f7d84663 9876b1193cfc564b1518d1f1b4459028ec75bf18
   > 7 75d9f02dfe2874aa938ee8c18fa27c1328cfb023 7371bc5168f70e1b7c8dbf7c8bedf9d79f51dd82
   > 
   > zzabc123 first
   > EOF
 
   $ hg tip
   changeset:   8:c4be69a18c11
-  $TESTTMP/t/.hg/cache/tags, line 1: node 'zzabc123' is not well formed
-  .hg/cache/tags is corrupt; rebuilding it
+  $TESTTMP/t/.hg/cache/tags1-visible, line 1: node 'zzabc123' is not well formed
+  .hg/cache/tags1-visible is corrupt; rebuilding it
   .hgtags@75d9f02dfe28, line 2: cannot parse entry
   .hgtags@75d9f02dfe28, line 4: node 'foo' is not well formed
   .hgtags@c4be69a18c11, line 2: node 'x' is not well formed
   tag:         tip
@@ -198,9 +198,9 @@  should be written automatically.
   date:        Thu Jan 01 00:00:00 1970 +0000
   summary:     head
   
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags1-visible
   8 c4be69a18c11e8bc3a5fdbb576017c25f7d84663 9876b1193cfc564b1518d1f1b4459028ec75bf18
   7 75d9f02dfe2874aa938ee8c18fa27c1328cfb023 7371bc5168f70e1b7c8dbf7c8bedf9d79f51dd82
   
   acb14030fe0a21b60322c440ad2d20cf7685a376 first
@@ -262,9 +262,9 @@  Detailed dump of tag info:
   bbd179dfa0a71671c253b3ae0aa1513b60d199fa bar
 
 Dump cache:
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags1-visible
   4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d
   3 6fa450212aeb2a21ed616a54aea39a4a27894cd7 7d3b718c964ef37b89e550ebdafd5789e76ce1b0
   2 7a94127795a33c10a370c93f731fd9fea0b79af6 0c04f2a8af31de17fab7422878ee5a2dadbc943d
   
@@ -273,9 +273,9 @@  Dump cache:
   78391a272241d70354aa14c874552cad6b51bb42 bar
 
 External .hgtags filenode cache marker is handled
 
-  $ cat > .hg/cache/tags << EOF
+  $ cat > .hg/cache/tags1-visible << EOF
   > external 4 0c192d7d5e6b78a714de54a2e9627952a877e25a 2e21d3312350ce63785cda82526c951211e76bab
   > 
   > bbd179dfa0a71671c253b3ae0aa1513b60d199fa bar
   > bbd179dfa0a71671c253b3ae0aa1513b60d199fa bar
@@ -287,9 +287,9 @@  External .hgtags filenode cache marker i
   bar                                1:78391a272241
 
 We should get an old style cache again
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags1-visible
   4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d
   3 6fa450212aeb2a21ed616a54aea39a4a27894cd7 7d3b718c964ef37b89e550ebdafd5789e76ce1b0
   2 7a94127795a33c10a370c93f731fd9fea0b79af6 0c04f2a8af31de17fab7422878ee5a2dadbc943d
   
@@ -399,9 +399,9 @@  Strip 2: destroy whole branch, no old he
   saved backup bundle to $TESTTMP/t3/.hg/strip-backup/*-backup.hg (glob)
   $ hg tags                  # partly stale
   tip                                4:735c3ca72986
   bar                                0:bbd179dfa0a7
-  $ rm -f .hg/cache/tags
+  $ rm -f .hg/cache/tags1-visible
   $ hg tags                  # cold cache
   tip                                4:735c3ca72986
   bar                                0:bbd179dfa0a7