Patchwork [1,of,9,sparse] localrepo: add sparse caches

login
register
mail settings
Submitter Gregory Szorc
Date July 6, 2017, 9:54 p.m.
Message ID <56de1555b77f5cd55303.1499378057@ubuntu-vm-main>
Download mbox | patch
Permalink /patch/22053/
State Accepted
Headers show

Comments

Gregory Szorc - July 6, 2017, 9:54 p.m.
# HG changeset patch
# User Gregory Szorc <gregory.szorc@gmail.com>
# Date 1499368853 25200
#      Thu Jul 06 12:20:53 2017 -0700
# Node ID 56de1555b77f5cd553032c54bddb79645316c86d
# Parent  ca4b78eb11e7a67600e85784df4da2655351b6d2
localrepo: add sparse caches

The sparse extension maintains caches for the sparse files
to a signature and a signature to a matcher. This allows the
sparse matchers to be resolved quickly, which is apparently
something that can occur in loops.

This patch ports the sparse caches to the localrepo class
pretty much as-is. There is potentially room to improve the
caching mechanism. But that can be done as a follow-up.

The default invalidatecaches() now clears the relevant sparse
cache. invalidatesignaturecache() has been moved to sparse.py.
via Mercurial-devel - July 6, 2017, 10:08 p.m.
On Thu, Jul 6, 2017 at 2:54 PM, Gregory Szorc <gregory.szorc@gmail.com> wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc@gmail.com>
> # Date 1499368853 25200
> #      Thu Jul 06 12:20:53 2017 -0700
> # Node ID 56de1555b77f5cd553032c54bddb79645316c86d
> # Parent  ca4b78eb11e7a67600e85784df4da2655351b6d2
> localrepo: add sparse caches
>
> diff --git a/hgext/sparse.py b/hgext/sparse.py
> --- a/hgext/sparse.py
> +++ b/hgext/sparse.py
> @@ -421,7 +421,7 @@ def _wraprepo(ui, repo):
>              """Returns the signature string representing the contents of the
>              current project sparse configuration. This can be used to cache the
>              sparse matcher for a given set of revs."""
> -            signaturecache = self.signaturecache
> +            signaturecache = self._sparsesignaturecache
>              signature = signaturecache.get('signature')
>              if includetemp:
>                  tempsignature = signaturecache.get('tempsignature')
> @@ -523,7 +516,7 @@ def _wraprepo(ui, repo):
>                  '\n'.join(sorted(include)),
>                  '\n'.join(sorted(exclude)))
>              self.vfs.write("sparse", raw)
> -            self.invalidatesignaturecache()
> +            sparse.invalidatesignaturecache(self)

What's the point of invalidatesignaturecache() when hgext/sparse.py
clearly knows about repo._signaturecache anyway?
Gregory Szorc - July 6, 2017, 10:13 p.m.
On Thu, Jul 6, 2017 at 3:08 PM, Martin von Zweigbergk <martinvonz@google.com
> wrote:

> On Thu, Jul 6, 2017 at 2:54 PM, Gregory Szorc <gregory.szorc@gmail.com>
> wrote:
> > # HG changeset patch
> > # User Gregory Szorc <gregory.szorc@gmail.com>
> > # Date 1499368853 25200
> > #      Thu Jul 06 12:20:53 2017 -0700
> > # Node ID 56de1555b77f5cd553032c54bddb79645316c86d
> > # Parent  ca4b78eb11e7a67600e85784df4da2655351b6d2
> > localrepo: add sparse caches
> >
> > diff --git a/hgext/sparse.py b/hgext/sparse.py
> > --- a/hgext/sparse.py
> > +++ b/hgext/sparse.py
> > @@ -421,7 +421,7 @@ def _wraprepo(ui, repo):
> >              """Returns the signature string representing the contents
> of the
> >              current project sparse configuration. This can be used to
> cache the
> >              sparse matcher for a given set of revs."""
> > -            signaturecache = self.signaturecache
> > +            signaturecache = self._sparsesignaturecache
> >              signature = signaturecache.get('signature')
> >              if includetemp:
> >                  tempsignature = signaturecache.get('tempsignature')
> > @@ -523,7 +516,7 @@ def _wraprepo(ui, repo):
> >                  '\n'.join(sorted(include)),
> >                  '\n'.join(sorted(exclude)))
> >              self.vfs.write("sparse", raw)
> > -            self.invalidatesignaturecache()
> > +            sparse.invalidatesignaturecache(self)
>
> What's the point of invalidatesignaturecache() when hgext/sparse.py
> clearly knows about repo._signaturecache anyway?
>

It /could/ be inlined in sparse.py. Since caching is hard and I didn't want
to rewrite the world, I decided to cargo cult the standalone function
(which also increases readability IMO). I can inline in a follow-up if you
want.
via Mercurial-devel - July 6, 2017, 10:15 p.m.
On Thu, Jul 6, 2017 at 3:13 PM, Gregory Szorc <gregory.szorc@gmail.com> wrote:
> On Thu, Jul 6, 2017 at 3:08 PM, Martin von Zweigbergk
> <martinvonz@google.com> wrote:
>>
>> On Thu, Jul 6, 2017 at 2:54 PM, Gregory Szorc <gregory.szorc@gmail.com>
>> wrote:
>> > # HG changeset patch
>> > # User Gregory Szorc <gregory.szorc@gmail.com>
>> > # Date 1499368853 25200
>> > #      Thu Jul 06 12:20:53 2017 -0700
>> > # Node ID 56de1555b77f5cd553032c54bddb79645316c86d
>> > # Parent  ca4b78eb11e7a67600e85784df4da2655351b6d2
>> > localrepo: add sparse caches
>> >
>> > diff --git a/hgext/sparse.py b/hgext/sparse.py
>> > --- a/hgext/sparse.py
>> > +++ b/hgext/sparse.py
>> > @@ -421,7 +421,7 @@ def _wraprepo(ui, repo):
>> >              """Returns the signature string representing the contents
>> > of the
>> >              current project sparse configuration. This can be used to
>> > cache the
>> >              sparse matcher for a given set of revs."""
>> > -            signaturecache = self.signaturecache
>> > +            signaturecache = self._sparsesignaturecache
>> >              signature = signaturecache.get('signature')
>> >              if includetemp:
>> >                  tempsignature = signaturecache.get('tempsignature')
>> > @@ -523,7 +516,7 @@ def _wraprepo(ui, repo):
>> >                  '\n'.join(sorted(include)),
>> >                  '\n'.join(sorted(exclude)))
>> >              self.vfs.write("sparse", raw)
>> > -            self.invalidatesignaturecache()
>> > +            sparse.invalidatesignaturecache(self)
>>
>> What's the point of invalidatesignaturecache() when hgext/sparse.py
>> clearly knows about repo._signaturecache anyway?
>
>
> It /could/ be inlined in sparse.py. Since caching is hard and I didn't want
> to rewrite the world, I decided to cargo cult the standalone function (which
> also increases readability IMO). I can inline in a follow-up if you want.
>

Yes, please do. Either a follow-up that inlines it or one that does a
better job of encapsulating it. I'm not sure which I'd prefer, so
inlining seems fine for now at least.

Patch

diff --git a/hgext/sparse.py b/hgext/sparse.py
--- a/hgext/sparse.py
+++ b/hgext/sparse.py
@@ -421,7 +421,7 @@  def _wraprepo(ui, repo):
             """Returns the signature string representing the contents of the
             current project sparse configuration. This can be used to cache the
             sparse matcher for a given set of revs."""
-            signaturecache = self.signaturecache
+            signaturecache = self._sparsesignaturecache
             signature = signaturecache.get('signature')
             if includetemp:
                 tempsignature = signaturecache.get('tempsignature')
@@ -445,13 +445,6 @@  def _wraprepo(ui, repo):
                     signaturecache['tempsignature'] = tempsignature
             return '%s %s' % (str(signature), str(tempsignature))
 
-        def invalidatecaches(self):
-            self.invalidatesignaturecache()
-            return super(SparseRepo, self).invalidatecaches()
-
-        def invalidatesignaturecache(self):
-            self.signaturecache.clear()
-
         def sparsematch(self, *revs, **kwargs):
             """Returns the sparse match function for the given revs.
 
@@ -470,7 +463,7 @@  def _wraprepo(ui, repo):
 
             key = '%s %s' % (str(signature), ' '.join([str(r) for r in revs]))
 
-            result = self.sparsecache.get(key, None)
+            result = self._sparsematchercache.get(key, None)
             if result:
                 return result
 
@@ -513,7 +506,7 @@  def _wraprepo(ui, repo):
                 tempincludes = self.gettemporaryincludes()
                 result = forceincludematcher(result, tempincludes)
 
-            self.sparsecache[key] = result
+            self._sparsematchercache[key] = result
 
             return result
 
@@ -523,7 +516,7 @@  def _wraprepo(ui, repo):
                 '\n'.join(sorted(include)),
                 '\n'.join(sorted(exclude)))
             self.vfs.write("sparse", raw)
-            self.invalidatesignaturecache()
+            sparse.invalidatesignaturecache(self)
 
         def addtemporaryincludes(self, files):
             includes = self.gettemporaryincludes()
@@ -541,7 +534,7 @@  def _wraprepo(ui, repo):
         def _writetemporaryincludes(self, includes):
             raw = '\n'.join(sorted(includes))
             self.vfs.write('tempsparse', raw)
-            self.invalidatesignaturecache()
+            sparse.invalidatesignaturecache(self)
 
         def prunetemporaryincludes(self):
             if repo.vfs.exists('tempsparse'):
@@ -572,15 +565,14 @@  def _wraprepo(ui, repo):
                     dirstate.drop(file)
 
                 self.vfs.unlink('tempsparse')
-                self.invalidatesignaturecache()
+                sparse.invalidatesignaturecache(self)
                 msg = _("cleaned up %d temporarily added file(s) from the "
                         "sparse checkout\n")
                 ui.status(msg % len(tempincludes))
 
     if 'dirstate' in repo._filecache:
         repo.dirstate.repo = repo
-    repo.sparsecache = {}
-    repo.signaturecache = {}
+
     repo.__class__ = SparseRepo
 
 @command('^debugsparse', [
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -422,6 +422,11 @@  class localrepository(object):
         # generic mapping between names and nodes
         self.names = namespaces.namespaces()
 
+        # Key to signature value.
+        self._sparsesignaturecache = {}
+        # Signature to cached matcher instance.
+        self._sparsematchercache = {}
+
     def close(self):
         self._writecaches()
 
@@ -1300,6 +1305,7 @@  class localrepository(object):
 
         self.unfiltered()._branchcaches.clear()
         self.invalidatevolatilesets()
+        self._sparsesignaturecache.clear()
 
     def invalidatevolatilesets(self):
         self.filteredrevcache.clear()
diff --git a/mercurial/sparse.py b/mercurial/sparse.py
--- a/mercurial/sparse.py
+++ b/mercurial/sparse.py
@@ -126,3 +126,6 @@  def activeprofiles(repo):
         profiles.update(patternsforrev(repo, rev)[2])
 
     return profiles
+
+def invalidatesignaturecache(repo):
+    repo._sparsesignaturecache.clear()