Patchwork [3,of,4,V2] lfs: override walk() in lfsvfs

login
register
mail settings
Submitter Matt Harbison
Date Dec. 10, 2017, 1:57 a.m.
Message ID <f060232e47d7e6f480c1.1512871030@Envy>
Download mbox | patch
Permalink /patch/26157/
State Accepted
Headers show

Comments

Matt Harbison - Dec. 10, 2017, 1:57 a.m.
# HG changeset patch
# User Matt Harbison <matt_harbison@yahoo.com>
# Date 1512708246 18000
#      Thu Dec 07 23:44:06 2017 -0500
# Node ID f060232e47d7e6f480c102e091e444d5f79de889
# Parent  174a9429a46635c56e038def4a1ac87d8d96ba88
lfs: override walk() in lfsvfs

In order to fix the missing lfs store after an upgrade, I attempted to walk the
store vfs to hardlink to the upgraded repo's store.  But the custom join()
clashes with the default walk() implementation.  First, 'path=None' blew up in
the regex matcher, because it wanted a string.  But even if that is fixed, the
join to walk the root of the vfs wouldn't match the required xx/xx...xx pattern.

The first cut of this was a copy/paste/tweak of the base implementation, but
this version of walk() hides the internal directories, and treats the vfs as a
flat store.  I think this makes sense because most vfs methods call join() on
input paths, which wants the simple oid format.  It also relieves the caller
from having to deal with bogus files/directories in the store.
Yuya Nishihara - Dec. 12, 2017, 2:19 p.m.
On Sat, 09 Dec 2017 20:57:10 -0500, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison <matt_harbison@yahoo.com>
> # Date 1512708246 18000
> #      Thu Dec 07 23:44:06 2017 -0500
> # Node ID f060232e47d7e6f480c102e091e444d5f79de889
> # Parent  174a9429a46635c56e038def4a1ac87d8d96ba88
> lfs: override walk() in lfsvfs

> --- a/hgext/lfs/blobstore.py
> +++ b/hgext/lfs/blobstore.py
> @@ -15,6 +15,7 @@
>  
>  from mercurial import (
>      error,
> +    pathutil,
>      url as urlmod,
>      util,
>      vfs as vfsmod,
> @@ -32,6 +33,28 @@
>              raise error.ProgrammingError('unexpected lfs path: %s' % path)
>          return super(lfsvfs, self).join(path[0:2], path[2:])
>  
> +    def walk(self, path=None, onerror=None):
> +        """Yield (dirpath, '', oids) tuple for blobs under path
> +
> +        Oids only exist in the root of this vfs, so dirpath is always ''.
> +        """
> +        root = os.path.normpath(self.base)
> +        # when dirpath == root, dirpath[prefixlen:] becomes empty
> +        # because len(dirpath) < prefixlen.
> +        prefixlen = len(pathutil.normasprefix(root))
> +        oids = []
> +
> +        for dirpath, dirs, files in os.walk(self.reljoin(self.base, path or ''),
> +                                            onerror=onerror):
> +            dirpath = dirpath[prefixlen:]
> +
> +            # Silently skip unexpected files and directories
> +            if len(dirpath) == 2:
> +                oids.extend([dirpath + f for f in files
> +                             if _lfsre.match(dirpath + f)])

Accumulating all files might not be ideal as the directory structure is
designed to not, but I have no idea how lfsvfs.walk() should behave.

> +
> +        yield ('', '', oids)
                      ^^
'dirs' should be an empty list. Can you send a follow up?

Patch

diff --git a/hgext/lfs/blobstore.py b/hgext/lfs/blobstore.py
--- a/hgext/lfs/blobstore.py
+++ b/hgext/lfs/blobstore.py
@@ -15,6 +15,7 @@ 
 
 from mercurial import (
     error,
+    pathutil,
     url as urlmod,
     util,
     vfs as vfsmod,
@@ -32,6 +33,28 @@ 
             raise error.ProgrammingError('unexpected lfs path: %s' % path)
         return super(lfsvfs, self).join(path[0:2], path[2:])
 
+    def walk(self, path=None, onerror=None):
+        """Yield (dirpath, '', oids) tuple for blobs under path
+
+        Oids only exist in the root of this vfs, so dirpath is always ''.
+        """
+        root = os.path.normpath(self.base)
+        # when dirpath == root, dirpath[prefixlen:] becomes empty
+        # because len(dirpath) < prefixlen.
+        prefixlen = len(pathutil.normasprefix(root))
+        oids = []
+
+        for dirpath, dirs, files in os.walk(self.reljoin(self.base, path or ''),
+                                            onerror=onerror):
+            dirpath = dirpath[prefixlen:]
+
+            # Silently skip unexpected files and directories
+            if len(dirpath) == 2:
+                oids.extend([dirpath + f for f in files
+                             if _lfsre.match(dirpath + f)])
+
+        yield ('', '', oids)
+
 class filewithprogress(object):
     """a file-like object that supports __len__ and read.