Patchwork [4,of,4,v2] treemanifests: fix streaming clone

login
register
mail settings
Submitter Martin von Zweigbergk
Date Feb. 3, 2016, 6:34 a.m.
Message ID <229c08d23c42a6263ad4.1454481273@waste.org>
Download mbox | patch
Permalink /patch/12950/
State Superseded
Headers show

Comments

Martin von Zweigbergk - Feb. 3, 2016, 6:34 a.m.
# HG changeset patch
# User Martin von Zweigbergk <martinvonz@google.com>
# Date 1454457679 28800
#      Tue Feb 02 16:01:19 2016 -0800
# Node ID 229c08d23c42a6263ad42bad197c66420babd50a
# Parent  8f1c2d81ff45633457a1ba8b8f91d5bd347a0605
treemanifests: fix streaming clone

Similar to the previous patch, the .hg/store/meta/ directory does not
get copied when when using "hg clone --uncompressed". Fix by making
sure that the files in the directory are generated by store.walk().

Also update test-treemanifest.t to use an uppercase directory name so
we test that the path encoding works.
Martin von Zweigbergk - Feb. 3, 2016, 5:21 p.m.
On Tue, Feb 2, 2016 at 10:34 PM, Martin von Zweigbergk
<martinvonz@google.com> wrote:
> +  $ hg -R deeprepo debugcreatestreamclonebundle repo-packed.hg
> +  $ hg debugbundle --spec repo-packed.hg

Sorry! I forgot to amend the changes from "run-tests.py -i". This last
part should read as follows. I'm guessing it's easiest if the person
queuing this fixes that in flight.

+  $ hg -R deeprepo debugcreatestreamclonebundle repo-packed.hg
+  writing 3349 bytes for 18 files
+  bundle requirements: generaldelta, revlogv1, treemanifest
+  $ hg debugbundle --spec repo-packed.hg
+  none-packed1;requirements%3Dgeneraldelta%2Crevlogv1%2Ctreemanifest
Martin von Zweigbergk - Feb. 4, 2016, 12:41 a.m.
On Tue, Feb 2, 2016 at 10:34 PM, Martin von Zweigbergk
<martinvonz@google.com> wrote:
> # HG changeset patch
> # User Martin von Zweigbergk <martinvonz@google.com>
> # Date 1454457679 28800
> #      Tue Feb 02 16:01:19 2016 -0800
> # Node ID 229c08d23c42a6263ad42bad197c66420babd50a
> # Parent  8f1c2d81ff45633457a1ba8b8f91d5bd347a0605
> treemanifests: fix streaming clone
>
> Similar to the previous patch, the .hg/store/meta/ directory does not
> get copied when when using "hg clone --uncompressed". Fix by making
> sure that the files in the directory are generated by store.walk().
>
> Also update test-treemanifest.t to use an uppercase directory name so
> we test that the path encoding works.
>
> diff --git a/mercurial/store.py b/mercurial/store.py
> --- a/mercurial/store.py
> +++ b/mercurial/store.py
> @@ -332,6 +332,9 @@
>      def datafiles(self):
>          return self._walk('data', True)
>
> +    def metafiles(self):
> +        pass

And this should apparently have been "return []". That's more bugs
that I'd ask anyone to fix in flight. I'll resend this after my patch
to "hg verify" is in instead (it also conflicts with that). Sorry
about the noise.

Patch

diff --git a/mercurial/store.py b/mercurial/store.py
--- a/mercurial/store.py
+++ b/mercurial/store.py
@@ -332,6 +332,9 @@ 
     def datafiles(self):
         return self._walk('data', True)
 
+    def metafiles(self):
+        pass
+
     def topfiles(self):
         # yield manifest before changelog
         return reversed(self._walk('', False))
@@ -341,6 +344,8 @@ 
         # yield data files first
         for x in self.datafiles():
             yield x
+        for x in self.metafiles():
+            yield x
         for x in self.topfiles():
             yield x
 
@@ -503,6 +508,14 @@ 
                 if err.errno != errno.ENOENT:
                     raise
 
+    def metafiles(self):
+        for a, b, size in self._walk('meta', True):
+            try:
+                a = decodefilename(a)
+            except KeyError:
+                a = None
+            yield a, b, size
+
     def copylist(self):
         d = ('data meta dh fncache phaseroots obsstore'
              ' 00manifest.d 00manifest.i 00changelog.d 00changelog.i')
diff --git a/tests/test-treemanifest.t b/tests/test-treemanifest.t
--- a/tests/test-treemanifest.t
+++ b/tests/test-treemanifest.t
@@ -367,7 +367,7 @@ 
   $ hg --config experimental.treemanifest=True init deeprepo
   $ cd deeprepo
 
-  $ mkdir a
+  $ mkdir A
   $ mkdir b
   $ mkdir b/bar
   $ mkdir b/bar/orange
@@ -376,8 +376,8 @@ 
   $ mkdir b/foo/apple
   $ mkdir b/foo/apple/bees
 
-  $ touch a/one.txt
-  $ touch a/two.txt
+  $ touch A/one.txt
+  $ touch A/two.txt
   $ touch b/bar/fruits.txt
   $ touch b/bar/orange/fly/gnat.py
   $ touch b/bar/orange/fly/housefly.txt
@@ -393,8 +393,8 @@ 
 Test files from the root.
 
   $ hg files -r .
-  a/one.txt (glob)
-  a/two.txt (glob)
+  A/one.txt (glob)
+  A/two.txt (glob)
   b/bar/fruits.txt (glob)
   b/bar/orange/fly/gnat.py (glob)
   b/bar/orange/fly/housefly.txt (glob)
@@ -411,56 +411,56 @@ 
 
 Test files for a subdirectory.
 
-  $ mv .hg/store/meta/a oldmf
+  $ mv .hg/store/meta/_a oldmf
   $ hg files -r . b
   b/bar/fruits.txt (glob)
   b/bar/orange/fly/gnat.py (glob)
   b/bar/orange/fly/housefly.txt (glob)
   b/foo/apple/bees/flower.py (glob)
-  $ mv oldmf .hg/store/meta/a
+  $ mv oldmf .hg/store/meta/_a
 
 Test files with just includes and excludes.
 
-  $ mv .hg/store/meta/a oldmf
+  $ mv .hg/store/meta/_a oldmf
   $ mv .hg/store/meta/b/bar/orange/fly oldmf2
   $ mv .hg/store/meta/b/foo/apple/bees oldmf3
   $ hg files -r . -I path:b/bar -X path:b/bar/orange/fly -I path:b/foo -X path:b/foo/apple/bees
   b/bar/fruits.txt (glob)
-  $ mv oldmf .hg/store/meta/a
+  $ mv oldmf .hg/store/meta/_a
   $ mv oldmf2 .hg/store/meta/b/bar/orange/fly
   $ mv oldmf3 .hg/store/meta/b/foo/apple/bees
 
 Test files for a subdirectory, excluding a directory within it.
 
-  $ mv .hg/store/meta/a oldmf
+  $ mv .hg/store/meta/_a oldmf
   $ mv .hg/store/meta/b/foo oldmf2
   $ hg files -r . -X path:b/foo b
   b/bar/fruits.txt (glob)
   b/bar/orange/fly/gnat.py (glob)
   b/bar/orange/fly/housefly.txt (glob)
-  $ mv oldmf .hg/store/meta/a
+  $ mv oldmf .hg/store/meta/_a
   $ mv oldmf2 .hg/store/meta/b/foo
 
 Test files for a sub directory, including only a directory within it, and
 including an unrelated directory.
 
-  $ mv .hg/store/meta/a oldmf
+  $ mv .hg/store/meta/_a oldmf
   $ mv .hg/store/meta/b/foo oldmf2
   $ hg files -r . -I path:b/bar/orange -I path:a b
   b/bar/orange/fly/gnat.py (glob)
   b/bar/orange/fly/housefly.txt (glob)
-  $ mv oldmf .hg/store/meta/a
+  $ mv oldmf .hg/store/meta/_a
   $ mv oldmf2 .hg/store/meta/b/foo
 
 Test files for a pattern, including a directory, and excluding a directory
 within that.
 
-  $ mv .hg/store/meta/a oldmf
+  $ mv .hg/store/meta/_a oldmf
   $ mv .hg/store/meta/b/foo oldmf2
   $ mv .hg/store/meta/b/bar/orange oldmf3
   $ hg files -r . glob:**.txt -I path:b/bar -X path:b/bar/orange
   b/bar/fruits.txt (glob)
-  $ mv oldmf .hg/store/meta/a
+  $ mv oldmf .hg/store/meta/_a
   $ mv oldmf2 .hg/store/meta/b/foo
   $ mv oldmf3 .hg/store/meta/b/bar/orange
 
@@ -493,8 +493,8 @@ 
 Tree manifest revlogs exist.
   $ find deepclone/.hg/store/meta | sort
   deepclone/.hg/store/meta
-  deepclone/.hg/store/meta/a
-  deepclone/.hg/store/meta/a/00manifest.i
+  deepclone/.hg/store/meta/_a
+  deepclone/.hg/store/meta/_a/00manifest.i
   deepclone/.hg/store/meta/b
   deepclone/.hg/store/meta/b/00manifest.i
   deepclone/.hg/store/meta/b/bar
@@ -549,3 +549,24 @@ 
   crosschecking files in changesets and manifests
   checking files
   8 files, 3 changesets, 10 total revisions
+
+Stream clone works
+
+  $ hg clone --config experimental.changegroup3=True --uncompressed -U \
+  >   http://localhost:$HGPORT2 stream-clone
+  streaming all changes
+  18 files to transfer, * of data (glob)
+  transferred * in * seconds (*) (glob)
+  searching for changes
+  no changes found
+  $ hg -R stream-clone verify
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  8 files, 3 changesets, 10 total revisions
+
+Packed bundle works
+
+  $ hg -R deeprepo debugcreatestreamclonebundle repo-packed.hg
+  $ hg debugbundle --spec repo-packed.hg