Patchwork [1,of,3] bundle: allow creation of changegroup3 bundles

login
register
mail settings
Submitter via Mercurial-devel
Date May 4, 2017, 5:04 p.m.
Message ID <f4b255ff67d05e329cc3.1493917451@martinvonz.svl.corp.google.com>
Download mbox | patch
Permalink /patch/20431/
State Changes Requested
Headers show

Comments

via Mercurial-devel - May 4, 2017, 5:04 p.m.
# HG changeset patch
# User Martin von Zweigbergk <martinvonz@google.com>
# Date 1458952398 25200
#      Fri Mar 25 17:33:18 2016 -0700
# Node ID f4b255ff67d05e329cc3d80ce1552f30005f58e9
# Parent  06058cbb9209df54bfa0aaf9f7d10b836a1e3ee9
bundle: allow creation of changegroup3 bundles

With this change, one can use "hg bundle -t v3" to creat changegroup3
bundles, which support treemanifests and revlog flags. In treemanifest
repos, v3 is the default bundle format.

Note that although changegroup3 is experimental, this patch does not
try to hide the v3 format (but also doesn't document it). It's simpler
this way and it will still fail unless the user explicitly enables
changegroup3 (see test case).

Also note that bundlerepo still does not support treemanifests well,
so this patch will allow saving changesets to a bundle and unbundling
from it, but e.g. "hg log -p repo.bundle" will not work ("hg log
repo.bundle" will, though, since it doesn't involve reading
manifests). Pulling from it will also not work.
Pierre-Yves David - May 4, 2017, 7:14 p.m.
On 05/04/2017 07:04 PM, Martin von Zweigbergk via Mercurial-devel wrote:
> # HG changeset patch
> # User Martin von Zweigbergk <martinvonz@google.com>
> # Date 1458952398 25200
> #      Fri Mar 25 17:33:18 2016 -0700
> # Node ID f4b255ff67d05e329cc3d80ce1552f30005f58e9
> # Parent  06058cbb9209df54bfa0aaf9f7d10b836a1e3ee9
> bundle: allow creation of changegroup3 bundles
>
> With this change, one can use "hg bundle -t v3" to creat changegroup3
> bundles, which support treemanifests and revlog flags. In treemanifest
> repos, v3 is the default bundle format.

The bundle spec "version" cover a collection of setting for the 
underlying content. It is not directly correlated to the changegroup 
version.

Tree manifest is still experimental so it seems a bit early to burn a 
new bundle spec version on this especially since we have other bundle 
improvement coming (phase, tag cache, obsmarkers, etc).

So I don't think we should introduce a new bundlespec version yet. 
Instead I would recommend the following:

1) introduce a parameter to control de cg-version (cgversion=03?)
2) automatically bump the cg version to '03' if no spec is passed and 
tree manifest is used,
3) late in the 4.3 cycles consider introducing a v3 with all the new 
improvements (if some make sense at the user-visible feature level).

Cheers,
Gregory Szorc - May 4, 2017, 10:14 p.m.
On Thu, May 4, 2017 at 12:14 PM, Pierre-Yves David <
pierre-yves.david@ens-lyon.org> wrote:

>
>
> On 05/04/2017 07:04 PM, Martin von Zweigbergk via Mercurial-devel wrote:
>
>> # HG changeset patch
>> # User Martin von Zweigbergk <martinvonz@google.com>
>> # Date 1458952398 25200
>> #      Fri Mar 25 17:33:18 2016 -0700
>> # Node ID f4b255ff67d05e329cc3d80ce1552f30005f58e9
>> # Parent  06058cbb9209df54bfa0aaf9f7d10b836a1e3ee9
>> bundle: allow creation of changegroup3 bundles
>>
>> With this change, one can use "hg bundle -t v3" to creat changegroup3
>> bundles, which support treemanifests and revlog flags. In treemanifest
>> repos, v3 is the default bundle format.
>>
>
> The bundle spec "version" cover a collection of setting for the underlying
> content. It is not directly correlated to the changegroup version.
>
> Tree manifest is still experimental so it seems a bit early to burn a new
> bundle spec version on this especially since we have other bundle
> improvement coming (phase, tag cache, obsmarkers, etc).
>
> So I don't think we should introduce a new bundlespec version yet. Instead
> I would recommend the following:
>
> 1) introduce a parameter to control de cg-version (cgversion=03?)
> 2) automatically bump the cg version to '03' if no spec is passed and tree
> manifest is used,
> 3) late in the 4.3 cycles consider introducing a v3 with all the new
> improvements (if some make sense at the user-visible feature level).
>

There was confusion about this in IRC.

I agree with what Pierre-Yves said: while this will require a "v3" to
support cg3, it is still a bit too early in cg3's support cycle to
introduce a stable, user-facing bundle spec version. Until we have a better
API for controlling bundle generation via bundle specs, I think this should
be hacked in via an experimental config option while recycling "v2."
Alternatively, we can name the version "expcg3" or something less stable
and rename it (presumably to "v3") once the feature is stable. The latter
will make it much easier to test the feature and won't result in
accidentally producing "v2" bundles that legacy clients choke on.

The concern with the "vN" versions is that every Mercurial client with
knowledge of that version string has to be capable of handling any future
bundle generated with that same "vN" version (or have code for filtering
from other data in the clone bundles manifest, preferably from BUNDLESPEC).
This means every time there is a new required bundle part, we have to
invent a new "vN" or find some other way to alter clonebundles entries so
old clients know they can't consume them.

Patch

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -40,6 +40,7 @@ 
 # Maps bundle version human names to changegroup versions.
 _bundlespeccgversions = {'v1': '01',
                          'v2': '02',
+                         'v3': '03',
                          'packed1': 's1',
                          'bundle2': '02', #legacy
                         }
@@ -136,6 +137,9 @@ 
             # Modern compression engines require v2.
             if compression not in _bundlespecv1compengines:
                 version = 'v2'
+            # Treemanifests require v3
+            if 'treemanifest' in repo.requirements:
+                version = 'v3'
         elif spec in _bundlespeccgversions:
             if spec == 'packed1':
                 compression = 'none'
@@ -232,6 +236,8 @@ 
                 version = part.params['version']
                 if version in ('01', '02'):
                     version = 'v2'
+                elif version == '03':
+                    version = 'v3'
                 else:
                     raise error.Abort(_('changegroup version %s does not have '
                                         'a known bundlespec') % version,
diff --git a/tests/test-bundle-type.t b/tests/test-bundle-type.t
--- a/tests/test-bundle-type.t
+++ b/tests/test-bundle-type.t
@@ -2,6 +2,8 @@ 
   $ cat << EOF >> $HGRCPATH
   > [format]
   > usegeneraldelta=yes
+  > [experimental]
+  > changegroup3=yes
   > EOF
 
 bundle w/o type option
@@ -65,7 +67,8 @@ 
   >   cd ..
   > }
 
-  $ for t in "None" "bzip2" "gzip" "none-v2" "v2" "v1" "gzip-v1"; do
+  $ for t in "None" "bzip2" "gzip" "none-v3" "v3" "none-v2" "v2" "v1" \
+  >     "gzip-v1"; do
   >   testbundle $t
   > done
   % test bundle type None
@@ -95,6 +98,24 @@ 
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
   gzip-v2
   
+  % test bundle type none-v3
+  searching for changes
+  1 changesets found
+  HG20\x00\x00 (esc)
+  Stream params: {}
+  changegroup -- "sortdict([('version', '03'), ('nbchanges', '1')])"
+      c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  none-v3
+  
+  % test bundle type v3
+  searching for changes
+  1 changesets found
+  HG20\x00\x00 (esc)
+  Stream params: sortdict([('Compression', 'BZ')])
+  changegroup -- "sortdict([('version', '03'), ('nbchanges', '1')])"
+      c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  bzip2-v3
+  
   % test bundle type none-v2
   searching for changes
   1 changesets found
@@ -127,6 +148,9 @@ 
   c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
   gzip-v1
   
+  $ hg --config experimental.changegroup3=no -R t1 bundle -t v3 bv3 tv3
+  abort: repository does not support bundle version 03
+  [255]
 
 Compression level can be adjusted for bundle2 bundles
 
diff --git a/tests/test-treemanifest.t b/tests/test-treemanifest.t
--- a/tests/test-treemanifest.t
+++ b/tests/test-treemanifest.t
@@ -867,3 +867,23 @@ 
   1:678d3574b88c
   1:678d3574b88c
   $ hg --config extensions.strip= strip -r . -q
+
+Bundle
+
+  $ cd ..
+  $ hg -R deeprepo bundle --all deeprepo.bundle
+  4 changesets found
+  $ hg debugbundle deeprepo.bundle
+  Stream params: sortdict([('Compression', 'BZ')])
+  changegroup -- "sortdict([('version', '03'), ('nbchanges', '4')])"
+      775704be6f529df4abf056a4644c866bd6dd6d3e
+      726ec8fc7e58ec32b0049698008851c6040abfa2
+      5a87d5a8da90c45b84639c95eb188b70f5b40a4e
+      523e5c631710b290a8d536284a27b9a6eca18bcd
+  $ hg --config experimental.treemanifest=True init empty-repo2
+  $ hg unbundle -R empty-repo2 deeprepo.bundle
+  adding changesets
+  adding manifests
+  adding file changes
+  added 4 changesets with 18 changes to 8 files
+  (run 'hg update' to get a working copy)