Patchwork [5,of,8,sparse,V2] dirstate: expose a sparse matcher on dirstate (API)

login
register
mail settings
Submitter Gregory Szorc
Date July 11, 2017, 4:57 a.m.
Message ID <03eed39f7af2db1678c2.1499749024@ubuntu-vm-main>
Download mbox | patch
Permalink /patch/22217/
State Accepted
Headers show

Comments

Gregory Szorc - July 11, 2017, 4:57 a.m.
# HG changeset patch
# User Gregory Szorc <gregory.szorc@gmail.com>
# Date 1499555884 25200
#      Sat Jul 08 16:18:04 2017 -0700
# Node ID 03eed39f7af2db1678c2d39a2a7342e5c66b0804
# Parent  ee818b5e70b42f4bd65605c5872d06e5bc0a0bb5
dirstate: expose a sparse matcher on dirstate (API)

The sparse extension performs a lot of monkeypatching of dirstate
to make it sparse aware. Essentially, various operations need to
take the active sparse config into account. They do this by obtaining
a matcher representing the sparse config and filtering paths through
it.

The monkeypatching is done by stuffing a reference to a repo on
dirstate and calling sparse.matcher() (which takes a repo instance)
during each function call. The reason this function takes a repo
instance is because resolving the sparse config may require resolving
file contents from filelogs, and that requires a repo. (If the
current sparse config references "profile" files, the contents of
those files from the dirstate's parent revisions is resolved.)

I seem to recall people having strong opinions that the dirstate
object not have a reference to a repo. So copying what the sparse
extension does probably won't fly in core. Plus, the dirstate
modifications shouldn't require a full repo: they only need a matcher.
So there's no good reason to stuff a reference to the repo in
dirstate.

This commit exposes a sparse matcher to dirstate via a property that
when looked up will call a function that eventually calls
sparse.matcher(). The repo instance is bound in a closure, so it
isn't exposed to dirstate.

This approach is functionally similar to what the sparse extension does
today, except it hides the repo instance from dirstate. The approach
is not optimal because we have to call a proxy function and
sparse.matcher() on every property lookup. There is room to cache
the matcher instance in dirstate. After all, the matcher only changes
if the dirstate's parents change or if the sparse config changes. It
feels like we should be able to detect both events and update the
matcher when this occurs. But for now we preserve the existing
semantics so we can move the dirstate sparseness bits into core. Once
in core, refactoring becomes a bit easier since it will be clearer how
all these components interact.

The sparse extension has been updated to use the new property.
Because all references to the repo on dirstate have been removed,
the code for setting it has been removed.
via Mercurial-devel - July 11, 2017, 6:17 a.m.
On Mon, Jul 10, 2017 at 9:57 PM, Gregory Szorc <gregory.szorc@gmail.com> wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc@gmail.com>
> # Date 1499555884 25200
> #      Sat Jul 08 16:18:04 2017 -0700
> # Node ID 03eed39f7af2db1678c2d39a2a7342e5c66b0804
> # Parent  ee818b5e70b42f4bd65605c5872d06e5bc0a0bb5
> dirstate: expose a sparse matcher on dirstate (API)

I've queued up to here for now, thanks!

I don't mind queuing the _ensuresparsematch() patch too. I think
raising an exception there is probably the right thing. Want to do
that in a followup or resend (if you agree, of course)?

Patch

diff --git a/hgext/largefiles/lfutil.py b/hgext/largefiles/lfutil.py
--- a/hgext/largefiles/lfutil.py
+++ b/hgext/largefiles/lfutil.py
@@ -26,6 +26,7 @@  from mercurial import (
     node,
     pycompat,
     scmutil,
+    sparse,
     util,
     vfs as vfsmod,
 )
@@ -147,7 +148,8 @@  def openlfdirstate(ui, repo, create=True
     lfstoredir = longname
     opener = vfsmod.vfs(vfs.join(lfstoredir))
     lfdirstate = largefilesdirstate(opener, ui, repo.root,
-                                     repo.dirstate._validate)
+                                    repo.dirstate._validate,
+                                    lambda: sparse.matcher(repo))
 
     # If the largefiles dirstate does not exist, populate and create
     # it. This ensures that we create it on the first meaningful
diff --git a/hgext/sparse.py b/hgext/sparse.py
--- a/hgext/sparse.py
+++ b/hgext/sparse.py
@@ -82,7 +82,6 @@  from mercurial import (
     error,
     extensions,
     hg,
-    localrepo,
     match as matchmod,
     registrar,
     sparse,
@@ -106,13 +105,6 @@  def extsetup(ui):
     _setupadd(ui)
     _setupdirstate(ui)
 
-def reposetup(ui, repo):
-    if not util.safehasattr(repo, 'dirstate'):
-        return
-
-    if 'dirstate' in repo._filecache:
-        repo.dirstate.repo = repo
-
 def replacefilecache(cls, propname, replacement):
     """Replace a filecache property with a new class. This allows changing the
     cache invalidation condition."""
@@ -200,13 +192,6 @@  def _setupdirstate(ui):
     and to prevent modifications to files outside the checkout.
     """
 
-    def _dirstate(orig, repo):
-        dirstate = orig(repo)
-        dirstate.repo = repo
-        return dirstate
-    extensions.wrapfunction(
-        localrepo.localrepository.dirstate, 'func', _dirstate)
-
     # The atrocity below is needed to wrap dirstate._ignore. It is a cached
     # property, which means normal function wrapping doesn't work.
     class ignorewrapper(object):
@@ -217,10 +202,9 @@  def _setupdirstate(ui):
             self.sparsematch = None
 
         def __get__(self, obj, type=None):
-            repo = obj.repo
             origignore = self.orig.__get__(obj)
 
-            sparsematch = sparse.matcher(repo)
+            sparsematch = obj._sparsematcher
             if sparsematch.always():
                 return origignore
 
@@ -241,7 +225,7 @@  def _setupdirstate(ui):
 
     # dirstate.rebuild should not add non-matching files
     def _rebuild(orig, self, parent, allfiles, changedfiles=None):
-        matcher = sparse.matcher(self.repo)
+        matcher = self._sparsematcher
         if not matcher.always():
             allfiles = allfiles.matches(matcher)
             if changedfiles:
@@ -262,8 +246,7 @@  def _setupdirstate(ui):
              '`hg add -s <file>` to include file directory while adding')
     for func in editfuncs:
         def _wrapper(orig, self, *args):
-            repo = self.repo
-            sparsematch = sparse.matcher(repo)
+            sparsematch = self._sparsematcher
             if not sparsematch.always():
                 for f in args:
                     if (f is not None and not sparsematch(f) and
diff --git a/mercurial/dirstate.py b/mercurial/dirstate.py
--- a/mercurial/dirstate.py
+++ b/mercurial/dirstate.py
@@ -70,7 +70,7 @@  def nonnormalentries(dmap):
 
 class dirstate(object):
 
-    def __init__(self, opener, ui, root, validate):
+    def __init__(self, opener, ui, root, validate, sparsematchfn):
         '''Create a new dirstate object.
 
         opener is an open()-like callable that can be used to open the
@@ -80,6 +80,7 @@  class dirstate(object):
         self._opener = opener
         self._validate = validate
         self._root = root
+        self._sparsematchfn = sparsematchfn
         # ntpath.join(root, '') of Python 2.7.9 does not add sep if root is
         # UNC path pointing to root share (issue4557)
         self._rootdir = pathutil.normasprefix(root)
@@ -197,6 +198,19 @@  class dirstate(object):
             f[normcase(name)] = name
         return f
 
+    @property
+    def _sparsematcher(self):
+        """The matcher for the sparse checkout.
+
+        The working directory may not include every file from a manifest. The
+        matcher obtained by this property will match a path if it is to be
+        included in the working directory.
+        """
+        # TODO there is potential to cache this property. For now, the matcher
+        # is resolved on every access. (But the called function does use a
+        # cache to keep the lookup fast.)
+        return self._sparsematchfn()
+
     @repocache('branch')
     def _branch(self):
         try:
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -53,6 +53,7 @@  from . import (
     revset,
     revsetlang,
     scmutil,
+    sparse,
     store,
     subrepo,
     tags as tagsmod,
@@ -570,8 +571,10 @@  class localrepository(object):
 
     @repofilecache('dirstate')
     def dirstate(self):
+        sparsematchfn = lambda: sparse.matcher(self)
+
         return dirstate.dirstate(self.vfs, self.ui, self.root,
-                                 self._dirstatevalidate)
+                                 self._dirstatevalidate, sparsematchfn)
 
     def _dirstatevalidate(self, node):
         try: