Patchwork [4,of,4] treemanifests: fix streaming clone

login
register
mail settings
Submitter Martin von Zweigbergk
Date Feb. 3, 2016, 6:01 a.m.
Message ID <2431679901e4a4227a90.1454479317@waste.org>
Download mbox | patch
Permalink /patch/12946/
State Superseded
Headers show

Comments

Martin von Zweigbergk - Feb. 3, 2016, 6:01 a.m.
# HG changeset patch
# User Martin von Zweigbergk <martinvonz@google.com>
# Date 1454457679 28800
#      Tue Feb 02 16:01:19 2016 -0800
# Node ID 2431679901e4a4227a9051936991f7e7a22e5df0
# Parent  8f1c2d81ff45633457a1ba8b8f91d5bd347a0605
treemanifests: fix streaming clone

Similar to the previous patch, the .hg/store/meta/ directory does not
get copied when when using "hg clone --uncompressed". Fix by making
sure that the files in the directory are generated by store.walk().

Also update test-treemanifest.t to use an uppercase directory name so
we test that the path encoding works.
Gregory Szorc - Feb. 3, 2016, 6:16 a.m.
On Tue, Feb 2, 2016 at 10:01 PM, Martin von Zweigbergk <
martinvonz@google.com> wrote:

> # HG changeset patch
> # User Martin von Zweigbergk <martinvonz@google.com>
> # Date 1454457679 28800
> #      Tue Feb 02 16:01:19 2016 -0800
> # Node ID 2431679901e4a4227a9051936991f7e7a22e5df0
> # Parent  8f1c2d81ff45633457a1ba8b8f91d5bd347a0605
> treemanifests: fix streaming clone
>
> Similar to the previous patch, the .hg/store/meta/ directory does not
> get copied when when using "hg clone --uncompressed". Fix by making
> sure that the files in the directory are generated by store.walk().
>
> Also update test-treemanifest.t to use an uppercase directory name so
> we test that the path encoding works.
>

We may also want to add some tests for `hg debugcreatestreamclonebundle`
(see test-bundle.t and test-clonebundles.t). At some point, functionality
in that command needs to be rolled into `hg bundle`, so the effort for
testing it shouldn't be lost.


>
> diff --git a/mercurial/store.py b/mercurial/store.py
> --- a/mercurial/store.py
> +++ b/mercurial/store.py
> @@ -332,6 +332,9 @@
>      def datafiles(self):
>          return self._walk('data', True)
>
> +    def metafiles(self):
> +        pass
> +
>      def topfiles(self):
>          # yield manifest before changelog
>          return reversed(self._walk('', False))
> @@ -341,6 +344,8 @@
>          # yield data files first
>          for x in self.datafiles():
>              yield x
> +        for x in self.metafiles():
> +            yield x
>          for x in self.topfiles():
>              yield x
>
> @@ -503,6 +508,14 @@
>                  if err.errno != errno.ENOENT:
>                      raise
>
> +    def metafiles(self):
> +        for a, b, size in self._walk('meta', True):
> +            try:
> +                a = decodefilename(a)
> +            except KeyError:
> +                a = None
> +            yield a, b, size
> +
>      def copylist(self):
>          d = ('data meta dh fncache phaseroots obsstore'
>               ' 00manifest.d 00manifest.i 00changelog.d 00changelog.i')
> diff --git a/tests/test-treemanifest.t b/tests/test-treemanifest.t
> --- a/tests/test-treemanifest.t
> +++ b/tests/test-treemanifest.t
> @@ -367,7 +367,7 @@
>    $ hg --config experimental.treemanifest=True init deeprepo
>    $ cd deeprepo
>
> -  $ mkdir a
> +  $ mkdir A
>    $ mkdir b
>    $ mkdir b/bar
>    $ mkdir b/bar/orange
> @@ -376,8 +376,8 @@
>    $ mkdir b/foo/apple
>    $ mkdir b/foo/apple/bees
>
> -  $ touch a/one.txt
> -  $ touch a/two.txt
> +  $ touch A/one.txt
> +  $ touch A/two.txt
>    $ touch b/bar/fruits.txt
>    $ touch b/bar/orange/fly/gnat.py
>    $ touch b/bar/orange/fly/housefly.txt
> @@ -393,8 +393,8 @@
>  Test files from the root.
>
>    $ hg files -r .
> -  a/one.txt (glob)
> -  a/two.txt (glob)
> +  A/one.txt (glob)
> +  A/two.txt (glob)
>    b/bar/fruits.txt (glob)
>    b/bar/orange/fly/gnat.py (glob)
>    b/bar/orange/fly/housefly.txt (glob)
> @@ -411,56 +411,56 @@
>
>  Test files for a subdirectory.
>
> -  $ mv .hg/store/meta/a oldmf
> +  $ mv .hg/store/meta/_a oldmf
>    $ hg files -r . b
>    b/bar/fruits.txt (glob)
>    b/bar/orange/fly/gnat.py (glob)
>    b/bar/orange/fly/housefly.txt (glob)
>    b/foo/apple/bees/flower.py (glob)
> -  $ mv oldmf .hg/store/meta/a
> +  $ mv oldmf .hg/store/meta/_a
>
>  Test files with just includes and excludes.
>
> -  $ mv .hg/store/meta/a oldmf
> +  $ mv .hg/store/meta/_a oldmf
>    $ mv .hg/store/meta/b/bar/orange/fly oldmf2
>    $ mv .hg/store/meta/b/foo/apple/bees oldmf3
>    $ hg files -r . -I path:b/bar -X path:b/bar/orange/fly -I path:b/foo -X
> path:b/foo/apple/bees
>    b/bar/fruits.txt (glob)
> -  $ mv oldmf .hg/store/meta/a
> +  $ mv oldmf .hg/store/meta/_a
>    $ mv oldmf2 .hg/store/meta/b/bar/orange/fly
>    $ mv oldmf3 .hg/store/meta/b/foo/apple/bees
>
>  Test files for a subdirectory, excluding a directory within it.
>
> -  $ mv .hg/store/meta/a oldmf
> +  $ mv .hg/store/meta/_a oldmf
>    $ mv .hg/store/meta/b/foo oldmf2
>    $ hg files -r . -X path:b/foo b
>    b/bar/fruits.txt (glob)
>    b/bar/orange/fly/gnat.py (glob)
>    b/bar/orange/fly/housefly.txt (glob)
> -  $ mv oldmf .hg/store/meta/a
> +  $ mv oldmf .hg/store/meta/_a
>    $ mv oldmf2 .hg/store/meta/b/foo
>
>  Test files for a sub directory, including only a directory within it, and
>  including an unrelated directory.
>
> -  $ mv .hg/store/meta/a oldmf
> +  $ mv .hg/store/meta/_a oldmf
>    $ mv .hg/store/meta/b/foo oldmf2
>    $ hg files -r . -I path:b/bar/orange -I path:a b
>    b/bar/orange/fly/gnat.py (glob)
>    b/bar/orange/fly/housefly.txt (glob)
> -  $ mv oldmf .hg/store/meta/a
> +  $ mv oldmf .hg/store/meta/_a
>    $ mv oldmf2 .hg/store/meta/b/foo
>
>  Test files for a pattern, including a directory, and excluding a directory
>  within that.
>
> -  $ mv .hg/store/meta/a oldmf
> +  $ mv .hg/store/meta/_a oldmf
>    $ mv .hg/store/meta/b/foo oldmf2
>    $ mv .hg/store/meta/b/bar/orange oldmf3
>    $ hg files -r . glob:**.txt -I path:b/bar -X path:b/bar/orange
>    b/bar/fruits.txt (glob)
> -  $ mv oldmf .hg/store/meta/a
> +  $ mv oldmf .hg/store/meta/_a
>    $ mv oldmf2 .hg/store/meta/b/foo
>    $ mv oldmf3 .hg/store/meta/b/bar/orange
>
> @@ -493,8 +493,8 @@
>  Tree manifest revlogs exist.
>    $ find deepclone/.hg/store/meta | sort
>    deepclone/.hg/store/meta
> -  deepclone/.hg/store/meta/a
> -  deepclone/.hg/store/meta/a/00manifest.i
> +  deepclone/.hg/store/meta/_a
> +  deepclone/.hg/store/meta/_a/00manifest.i
>    deepclone/.hg/store/meta/b
>    deepclone/.hg/store/meta/b/00manifest.i
>    deepclone/.hg/store/meta/b/bar
> @@ -549,3 +549,19 @@
>    crosschecking files in changesets and manifests
>    checking files
>    8 files, 3 changesets, 10 total revisions
> +
> +Stream clone should work
> +
> +  $ hg clone --config experimental.changegroup3=True --uncompressed -U \
> +  >   http://localhost:$HGPORT2 stream-clone
> +  streaming all changes
> +  18 files to transfer, * of data (glob)
> +  transferred * in * seconds (*) (glob)
> +  searching for changes
> +  no changes found
> +  $ hg -R stream-clone verify
> +  checking changesets
> +  checking manifests
> +  crosschecking files in changesets and manifests
> +  checking files
> +  8 files, 3 changesets, 10 total revisions
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>
Martin von Zweigbergk - Feb. 3, 2016, 6:29 a.m.
On Tue, Feb 2, 2016 at 10:16 PM Gregory Szorc <gregory.szorc@gmail.com>
wrote:

> On Tue, Feb 2, 2016 at 10:01 PM, Martin von Zweigbergk <
> martinvonz@google.com> wrote:
>
>> # HG changeset patch
>> # User Martin von Zweigbergk <martinvonz@google.com>
>> # Date 1454457679 28800
>> #      Tue Feb 02 16:01:19 2016 -0800
>> # Node ID 2431679901e4a4227a9051936991f7e7a22e5df0
>> # Parent  8f1c2d81ff45633457a1ba8b8f91d5bd347a0605
>> treemanifests: fix streaming clone
>>
>> Similar to the previous patch, the .hg/store/meta/ directory does not
>> get copied when when using "hg clone --uncompressed". Fix by making
>> sure that the files in the directory are generated by store.walk().
>>
>> Also update test-treemanifest.t to use an uppercase directory name so
>> we test that the path encoding works.
>>
>
> We may also want to add some tests for `hg debugcreatestreamclonebundle`
> (see test-bundle.t and test-clonebundles.t). At some point, functionality
> in that command needs to be rolled into `hg bundle`, so the effort for
> testing it shouldn't be lost.
>

Good idea. I had no idea such a thing even existed. Resending shortly.

Patch

diff --git a/mercurial/store.py b/mercurial/store.py
--- a/mercurial/store.py
+++ b/mercurial/store.py
@@ -332,6 +332,9 @@ 
     def datafiles(self):
         return self._walk('data', True)
 
+    def metafiles(self):
+        pass
+
     def topfiles(self):
         # yield manifest before changelog
         return reversed(self._walk('', False))
@@ -341,6 +344,8 @@ 
         # yield data files first
         for x in self.datafiles():
             yield x
+        for x in self.metafiles():
+            yield x
         for x in self.topfiles():
             yield x
 
@@ -503,6 +508,14 @@ 
                 if err.errno != errno.ENOENT:
                     raise
 
+    def metafiles(self):
+        for a, b, size in self._walk('meta', True):
+            try:
+                a = decodefilename(a)
+            except KeyError:
+                a = None
+            yield a, b, size
+
     def copylist(self):
         d = ('data meta dh fncache phaseroots obsstore'
              ' 00manifest.d 00manifest.i 00changelog.d 00changelog.i')
diff --git a/tests/test-treemanifest.t b/tests/test-treemanifest.t
--- a/tests/test-treemanifest.t
+++ b/tests/test-treemanifest.t
@@ -367,7 +367,7 @@ 
   $ hg --config experimental.treemanifest=True init deeprepo
   $ cd deeprepo
 
-  $ mkdir a
+  $ mkdir A
   $ mkdir b
   $ mkdir b/bar
   $ mkdir b/bar/orange
@@ -376,8 +376,8 @@ 
   $ mkdir b/foo/apple
   $ mkdir b/foo/apple/bees
 
-  $ touch a/one.txt
-  $ touch a/two.txt
+  $ touch A/one.txt
+  $ touch A/two.txt
   $ touch b/bar/fruits.txt
   $ touch b/bar/orange/fly/gnat.py
   $ touch b/bar/orange/fly/housefly.txt
@@ -393,8 +393,8 @@ 
 Test files from the root.
 
   $ hg files -r .
-  a/one.txt (glob)
-  a/two.txt (glob)
+  A/one.txt (glob)
+  A/two.txt (glob)
   b/bar/fruits.txt (glob)
   b/bar/orange/fly/gnat.py (glob)
   b/bar/orange/fly/housefly.txt (glob)
@@ -411,56 +411,56 @@ 
 
 Test files for a subdirectory.
 
-  $ mv .hg/store/meta/a oldmf
+  $ mv .hg/store/meta/_a oldmf
   $ hg files -r . b
   b/bar/fruits.txt (glob)
   b/bar/orange/fly/gnat.py (glob)
   b/bar/orange/fly/housefly.txt (glob)
   b/foo/apple/bees/flower.py (glob)
-  $ mv oldmf .hg/store/meta/a
+  $ mv oldmf .hg/store/meta/_a
 
 Test files with just includes and excludes.
 
-  $ mv .hg/store/meta/a oldmf
+  $ mv .hg/store/meta/_a oldmf
   $ mv .hg/store/meta/b/bar/orange/fly oldmf2
   $ mv .hg/store/meta/b/foo/apple/bees oldmf3
   $ hg files -r . -I path:b/bar -X path:b/bar/orange/fly -I path:b/foo -X path:b/foo/apple/bees
   b/bar/fruits.txt (glob)
-  $ mv oldmf .hg/store/meta/a
+  $ mv oldmf .hg/store/meta/_a
   $ mv oldmf2 .hg/store/meta/b/bar/orange/fly
   $ mv oldmf3 .hg/store/meta/b/foo/apple/bees
 
 Test files for a subdirectory, excluding a directory within it.
 
-  $ mv .hg/store/meta/a oldmf
+  $ mv .hg/store/meta/_a oldmf
   $ mv .hg/store/meta/b/foo oldmf2
   $ hg files -r . -X path:b/foo b
   b/bar/fruits.txt (glob)
   b/bar/orange/fly/gnat.py (glob)
   b/bar/orange/fly/housefly.txt (glob)
-  $ mv oldmf .hg/store/meta/a
+  $ mv oldmf .hg/store/meta/_a
   $ mv oldmf2 .hg/store/meta/b/foo
 
 Test files for a sub directory, including only a directory within it, and
 including an unrelated directory.
 
-  $ mv .hg/store/meta/a oldmf
+  $ mv .hg/store/meta/_a oldmf
   $ mv .hg/store/meta/b/foo oldmf2
   $ hg files -r . -I path:b/bar/orange -I path:a b
   b/bar/orange/fly/gnat.py (glob)
   b/bar/orange/fly/housefly.txt (glob)
-  $ mv oldmf .hg/store/meta/a
+  $ mv oldmf .hg/store/meta/_a
   $ mv oldmf2 .hg/store/meta/b/foo
 
 Test files for a pattern, including a directory, and excluding a directory
 within that.
 
-  $ mv .hg/store/meta/a oldmf
+  $ mv .hg/store/meta/_a oldmf
   $ mv .hg/store/meta/b/foo oldmf2
   $ mv .hg/store/meta/b/bar/orange oldmf3
   $ hg files -r . glob:**.txt -I path:b/bar -X path:b/bar/orange
   b/bar/fruits.txt (glob)
-  $ mv oldmf .hg/store/meta/a
+  $ mv oldmf .hg/store/meta/_a
   $ mv oldmf2 .hg/store/meta/b/foo
   $ mv oldmf3 .hg/store/meta/b/bar/orange
 
@@ -493,8 +493,8 @@ 
 Tree manifest revlogs exist.
   $ find deepclone/.hg/store/meta | sort
   deepclone/.hg/store/meta
-  deepclone/.hg/store/meta/a
-  deepclone/.hg/store/meta/a/00manifest.i
+  deepclone/.hg/store/meta/_a
+  deepclone/.hg/store/meta/_a/00manifest.i
   deepclone/.hg/store/meta/b
   deepclone/.hg/store/meta/b/00manifest.i
   deepclone/.hg/store/meta/b/bar
@@ -549,3 +549,19 @@ 
   crosschecking files in changesets and manifests
   checking files
   8 files, 3 changesets, 10 total revisions
+
+Stream clone should work
+
+  $ hg clone --config experimental.changegroup3=True --uncompressed -U \
+  >   http://localhost:$HGPORT2 stream-clone
+  streaming all changes
+  18 files to transfer, * of data (glob)
+  transferred * in * seconds (*) (glob)
+  searching for changes
+  no changes found
+  $ hg -R stream-clone verify
+  checking changesets
+  checking manifests
+  crosschecking files in changesets and manifests
+  checking files
+  8 files, 3 changesets, 10 total revisions