Patchwork tags: have a different cache file per filter level

login
register
mail settings
Submitter Pierre-Yves David
Date April 15, 2015, 10:40 p.m.
Message ID <f70474328e960e44c2ec.1429137615@marginatus.alto.octopoid.net>
Download mbox | patch
Permalink /patch/8695/
State Accepted
Headers show

Comments

Pierre-Yves David - April 15, 2015, 10:40 p.m.
# HG changeset patch
# User Pierre-Yves David <pierre-yves.david@fb.com>
# Date 1429137274 14400
#      Wed Apr 15 18:34:34 2015 -0400
# Node ID f70474328e960e44c2ec7c0d7908f1091200c8f3
# Parent  f2fd087a75ef4bce92746a1f4cd03661946b999f
tags: have a different cache file per filter level

Currently whichever filter level ask for tags last will write the data on disk.
This create massive issue when tags are read for "visible" and "unfiltered"
on large and multi headed repository (like Mozilla central). See issue4550 for
details

Each cache level recomputes its own cache without direct collaboration but they
all share the same 'hgtagsfnodes' cache. And that is were most of the time is
spent.
Pierre-Yves David - April 16, 2015, 7 a.m.
On 04/15/2015 06:40 PM, Pierre-Yves David wrote:
> # HG changeset patch
> # User Pierre-Yves David <pierre-yves.david@fb.com>
> # Date 1429137274 14400
> #      Wed Apr 15 18:34:34 2015 -0400
> # Node ID f70474328e960e44c2ec7c0d7908f1091200c8f3
> # Parent  f2fd087a75ef4bce92746a1f4cd03661946b999f
> tags: have a different cache file per filter level

This one have been pushed to the clowncopter after Matt's IRL review.
Gregory Szorc - April 16, 2015, 1:07 p.m.
On Thu, Apr 16, 2015 at 3:00 AM, Pierre-Yves David <
pierre-yves.david@ens-lyon.org> wrote:

> On 04/15/2015 06:40 PM, Pierre-Yves David wrote:
>
>> # HG changeset patch
>> # User Pierre-Yves David <pierre-yves.david@fb.com>
>> # Date 1429137274 14400
>> #      Wed Apr 15 18:34:34 2015 -0400
>> # Node ID f70474328e960e44c2ec7c0d7908f1091200c8f3
>> # Parent  f2fd087a75ef4bce92746a1f4cd03661946b999f
>> tags: have a different cache file per filter level
>>
>
> This one have been pushed to the clowncopter after Matt's IRL review.
>

I'm not a fan of having tags cache ship in this state for 3.4.

Our plan was to split tags cache into multiple filters/files (this patch)
and then *immediately* do a breaking format change so filtered revs
invalidation is performed properly. i.e. the new files would be written in
the new format from day 0.

If just this patch ships in 3.4, we'll have to establish a new set of
per-filter tags cache files with a new format in 3.5.

I'll try to whip up the format change patch today. I would appreciate if
this first patch wasn't pushed until it is accompanied by the format change.
Pierre-Yves David - April 16, 2015, 2:59 p.m.
On 04/16/2015 09:07 AM, Gregory Szorc wrote:
> On Thu, Apr 16, 2015 at 3:00 AM, Pierre-Yves David
> <pierre-yves.david@ens-lyon.org <mailto:pierre-yves.david@ens-lyon.org>>
> wrote:
>
>     On 04/15/2015 06:40 PM, Pierre-Yves David wrote:
>
>         # HG changeset patch
>         # User Pierre-Yves David <pierre-yves.david@fb.com
>         <mailto:pierre-yves.david@fb.com>>
>         # Date 1429137274 14400
>         #      Wed Apr 15 18:34:34 2015 -0400
>         # Node ID f70474328e960e44c2ec7c0d7908f1__091200c8f3
>         # Parent  f2fd087a75ef4bce92746a1f4cd036__61946b999f
>         tags: have a different cache file per filter level
>
>
>     This one have been pushed to the clowncopter after Matt's IRL review.
>
>
> I'm not a fan of having tags cache ship in this state for 3.4.
>
> Our plan was to split tags cache into multiple filters/files (this
> patch) and then *immediately* do a breaking format change so filtered
> revs invalidation is performed properly. i.e. the new files would be
> written in the new format from day 0.
>
> If just this patch ships in 3.4, we'll have to establish a new set of
> per-filter tags cache files with a new format in 3.5.
>
> I'll try to whip up the format change patch today. I would appreciate if
> this first patch wasn't pushed until it is accompanied by the format change.

I do not see much drawback in having multiple format change. These are 
cache file and the most expensive part is in the hgtagfnodecache anyway.
Matt Harbison - April 21, 2015, 3:37 a.m.
On Wed, 15 Apr 2015 18:40:15 -0400, Pierre-Yves David  
<pierre-yves.david@ens-lyon.org> wrote:

> # HG changeset patch
> # User Pierre-Yves David <pierre-yves.david@fb.com>
> # Date 1429137274 14400
> #      Wed Apr 15 18:34:34 2015 -0400
> # Node ID f70474328e960e44c2ec7c0d7908f1091200c8f3
> # Parent  f2fd087a75ef4bce92746a1f4cd03661946b999f
> tags: have a different cache file per filter level
>
> Currently whichever filter level ask for tags last will write the data  
> on disk.
> This create massive issue when tags are read for "visible" and  
> "unfiltered"
> on large and multi headed repository (like Mozilla central). See  
> issue4550 for
> details
>
> Each cache level recomputes its own cache without direct collaboration  
> but they
> all share the same 'hgtagsfnodes' cache. And that is were most of the  
> time is
> spent.

Does this patch (in repo as b061a2049662) mean we can now resurrect this:

http://selenic.com/repo/hg//rev/b08af8f0ac01

changeset:   25398:b08af8f0ac01
user:        Matt Harbison <matt_harbison@yahoo.com>
date:        Wed Oct 01 20:26:33 2014 -0400
summary:     localrepo: don't reintroduce pruned tag entries when tagging

Patch

diff --git a/mercurial/tags.py b/mercurial/tags.py
--- a/mercurial/tags.py
+++ b/mercurial/tags.py
@@ -244,10 +244,17 @@  def _updatetags(filetags, tagtype, allta
         else:
             tagtypes[name] = tagtype
         ahist.extend([n for n in bhist if n not in ahist])
         alltags[name] = anode, ahist
 
+def _filename(repo):
+    """name of a tagcache file for a given repo or repoview"""
+    filename = 'cache/tags'
+    if repo.filtername:
+        filename = '%s-%s' % (filename, repo.filtername)
+    return filename
+
 def _readtagcache(ui, repo):
     '''Read the tag cache.
 
     Returns a tuple (heads, fnodes, cachetags, shouldwrite).
 
@@ -263,11 +270,11 @@  def _readtagcache(ui, repo):
     If the cache is not up to date, the caller is responsible for reading tag
     info from each returned head. (See findglobaltags().)
     '''
 
     try:
-        cachefile = repo.vfs('cache/tags', 'r')
+        cachefile = repo.vfs(_filename(repo), 'r')
         # force reading the file for static-http
         cachelines = iter(cachefile)
     except IOError:
         cachefile = None
 
@@ -358,11 +365,11 @@  def _readtagcache(ui, repo):
     # cachefnode to get to each .hgtags revision quickly.
     return (repoheads, cachefnode, None, True)
 
 def _writetagcache(ui, repo, heads, tagfnode, cachetags):
     try:
-        cachefile = repo.vfs('cache/tags', 'w', atomictemp=True)
+        cachefile = repo.vfs(_filename(repo), 'w', atomictemp=True)
     except (OSError, IOError):
         return
 
     ui.log('tagscache', 'writing tags cache file with %d heads and %d tags\n',
             len(heads), len(cachetags))
diff --git a/tests/test-mq.t b/tests/test-mq.t
--- a/tests/test-mq.t
+++ b/tests/test-mq.t
@@ -309,28 +309,28 @@  qpop
 
 
 qpush with dump of tag cache
 Dump the tag cache to ensure that it has exactly one head after qpush.
 
-  $ rm -f .hg/cache/tags
+  $ rm -f .hg/cache/tags-visible
   $ hg tags > /dev/null
 
-.hg/cache/tags (pre qpush):
+.hg/cache/tags-visible (pre qpush):
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags-visible
   1 [\da-f]{40} (re)
   
   $ hg qpush
   applying test.patch
   now at: test.patch
   $ hg phase -r qbase
   2: draft
   $ hg tags > /dev/null
 
-.hg/cache/tags (post qpush):
+.hg/cache/tags-visible (post qpush):
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags-visible
   2 [\da-f]{40} (re)
   
   $ checkundo qpush
   $ cd ..
 
diff --git a/tests/test-obsolete-tag-cache.t b/tests/test-obsolete-tag-cache.t
--- a/tests/test-obsolete-tag-cache.t
+++ b/tests/test-obsolete-tag-cache.t
@@ -34,11 +34,11 @@  Trigger tags cache population by doing s
   | o  1:5f97d42da03f  test tag
   |/
   o  0:55482a6fb4b1 test1 initial
   
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags-visible
   4 042eb6bfcc4909bad84a1cbf6eb1ddf0ab587d41
   3 c3cb30f2d2cd0aae008cc91a07876e3c5131fd22 b3bce87817fe7ac9dca2834366c1d7534c095cf1
   
   55482a6fb4b1881fa8f746fd52cf6f096bb21c89 test1
   d75775ffbc6bca1794d300f5571272879bd280da test2
@@ -60,11 +60,11 @@  repopulation
   
 
 .hgtags filenodes for hidden heads should be visible (issue4550)
 (currently broken)
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags-visible
   7 eb610439e10e0c6b296f97b59624c2e24fc59e30 b3bce87817fe7ac9dca2834366c1d7534c095cf1
   
   55482a6fb4b1881fa8f746fd52cf6f096bb21c89 test1
   d75775ffbc6bca1794d300f5571272879bd280da test2
 
diff --git a/tests/test-tags.t b/tests/test-tags.t
--- a/tests/test-tags.t
+++ b/tests/test-tags.t
@@ -7,11 +7,11 @@  setup
   > EOF
 
 Helper functions:
 
   $ cacheexists() {
-  >   [ -f .hg/cache/tags ] && echo "tag cache exists" || echo "no tag cache"
+  >   [ -f .hg/cache/tags-visible ] && echo "tag cache exists" || echo "no tag cache"
   > }
 
   $ fnodescacheexists() {
   >   [ -f .hg/cache/hgtagsfnodes1 ] && echo "fnodes cache exists" || echo "no fnodes cache"
   > }
@@ -54,11 +54,11 @@  No fnodes cache because .hgtags file doe
   $ fnodescacheexists
   no fnodes cache
 
 Try corrupting the cache
 
-  $ printf 'a b' > .hg/cache/tags
+  $ printf 'a b' > .hg/cache/tags-visible
   $ hg identify
   acb14030fe0a tip
   $ cacheexists
   tag cache exists
   $ fnodescacheexists
@@ -100,11 +100,11 @@  The cache should have an empty entry for
   0010: ff ff ff ff ff ff ff ff b9 15 46 36 26 b7 b4 a7 |..........F6&...|
   0020: 73 e0 9e e3 c5 2f 51 0e 19 e0 5e 1f f9 66 d8 59 |s..../Q...^..f.Y|
 
 Repeat with cold tag cache:
 
-  $ rm -f .hg/cache/tags .hg/cache/hgtagsfnodes1
+  $ rm -f .hg/cache/tags-visible .hg/cache/hgtagsfnodes1
   $ hg identify
   b9154636be93 tip
 
   $ fnodescacheexists
   fnodes cache exists
@@ -115,20 +115,20 @@  Repeat with cold tag cache:
   0020: 73 e0 9e e3 c5 2f 51 0e 19 e0 5e 1f f9 66 d8 59 |s..../Q...^..f.Y|
 
 And again, but now unable to write tag cache:
 
 #if unix-permissions
-  $ rm -f .hg/cache/tags .hg/cache/hgtagsfnodes1
+  $ rm -f .hg/cache/tags-visible .hg/cache/hgtagsfnodes1
   $ chmod 555 .hg/cache
   $ hg identify
   b9154636be93 tip
   $ chmod 755 .hg/cache
 #endif
 
 Tag cache debug info written to blackbox log
 
-  $ rm -f .hg/cache/tags .hg/cache/hgtagsfnodes1
+  $ rm -f .hg/cache/tags-visible .hg/cache/hgtagsfnodes1
   $ hg identify
   b9154636be93 tip
   $ hg blackbox -l 5
   1970/01/01 00:00:00 bob> identify
   1970/01/01 00:00:00 bob> writing 48 bytes to cache/hgtagsfnodes1
@@ -136,11 +136,11 @@  Tag cache debug info written to blackbox
   1970/01/01 00:00:00 bob> writing tags cache file with 1 heads and 1 tags
   1970/01/01 00:00:00 bob> identify exited 0 after ?.?? seconds (glob)
 
 Failure to acquire lock results in no write
 
-  $ rm -f .hg/cache/tags .hg/cache/hgtagsfnodes1
+  $ rm -f .hg/cache/tags-visible .hg/cache/hgtagsfnodes1
   $ echo 'foo:1' > .hg/wlock
   $ hg identify
   b9154636be93 tip
   $ hg blackbox -l 5
   1970/01/01 00:00:00 bob> identify
@@ -152,11 +152,11 @@  Failure to acquire lock results in no wr
   $ fnodescacheexists
   no fnodes cache
 
   $ rm .hg/wlock
 
-  $ rm -f .hg/cache/tags .hg/cache/hgtagsfnodes1
+  $ rm -f .hg/cache/tags-visible .hg/cache/hgtagsfnodes1
   $ hg identify
   b9154636be93 tip
 
 Create a branch:
 
@@ -309,11 +309,11 @@  Detailed dump of tag info:
   rev 4: .hgtags:
   bbd179dfa0a71671c253b3ae0aa1513b60d199fa bar
 
 Dump cache:
 
-  $ cat .hg/cache/tags
+  $ cat .hg/cache/tags-visible
   4 0c192d7d5e6b78a714de54a2e9627952a877e25a 0c04f2a8af31de17fab7422878ee5a2dadbc943d
   3 6fa450212aeb2a21ed616a54aea39a4a27894cd7 7d3b718c964ef37b89e550ebdafd5789e76ce1b0
   2 7a94127795a33c10a370c93f731fd9fea0b79af6 0c04f2a8af31de17fab7422878ee5a2dadbc943d
   
   bbd179dfa0a71671c253b3ae0aa1513b60d199fa bar
@@ -354,11 +354,11 @@  Errors writing to .hgtags fnodes cache a
 
   $ echo dummy2 > foo
   $ hg commit -m throwaway2
 
   $ chmod a-w .hg/cache/hgtagsfnodes1
-  $ rm -f .hg/cache/tags
+  $ rm -f .hg/cache/tags-visible
 
   $ hg tags
   tip                                6:b968051b5cf3
   bar                                1:78391a272241
 
@@ -370,11 +370,11 @@  Errors writing to .hgtags fnodes cache a
   1970/01/01 00:00:00 bob> tags exited 0 after * seconds (glob)
 
   $ chmod a+w .hg/cache/hgtagsfnodes1
 #endif
 
-  $ rm -f .hg/cache/tags
+  $ rm -f .hg/cache/tags-visible
   $ hg tags
   tip                                6:b968051b5cf3
   bar                                1:78391a272241
 
   $ hg blackbox -l 5
@@ -522,11 +522,11 @@  Strip 2: destroy whole branch, no old he
   $ hg --config extensions.mq= strip 4
   saved backup bundle to $TESTTMP/t3/.hg/strip-backup/*-backup.hg (glob)
   $ hg tags                  # partly stale
   tip                                4:735c3ca72986
   bar                                0:bbd179dfa0a7
-  $ rm -f .hg/cache/tags
+  $ rm -f .hg/cache/tags-visible
   $ hg tags                  # cold cache
   tip                                4:735c3ca72986
   bar                                0:bbd179dfa0a7
 
 Test tag rank with 3 heads: