Patchwork localrepo: send client's bundlecaps to a server during a pull

login
register
mail settings
Submitter Wojciech Lopata
Date Aug. 30, 2013, 8:14 p.m.
Message ID <b78c0dcf3d674c23462c.1377893642@dev1179.prn1.facebook.com>
Download mbox | patch
Permalink /patch/2290/
State Superseded
Headers show

Comments

Wojciech Lopata - Aug. 30, 2013, 8:14 p.m.
# HG changeset patch
# User Wojciech Lopata <lopek@fb.com>
# Date 1377891837 25200
#      Fri Aug 30 12:43:57 2013 -0700
# Node ID b78c0dcf3d674c23462c1b68dff43e6d3277f295
# Parent  73d761ed3dbddd06d4a65027ef82bb77ae0ec82f
localrepo: send client's bundlecaps to a server during a pull

Thanks to this change extensions that modify format of revlog and are used on
client and server side will be able to customize format of bundles sent during
a pull, and possibly make this operation faster.
Wojciech Lopata - Aug. 30, 2013, 8:18 p.m.
It is likely that this patch is bad, since I'm not sure if I know what bundlecaps truly are (and I make them a property of localrepo). I didn't find any piece of code that would be making use of this feature. I will appreciate someone taking a look and pointing me the right way of doing what I want to do.

My usecase: I change format of the manifest. To make pulls between two repos (one of which might not be using my extension) possible, I need to convert data stored in manifest to 'regular' format before bundling. This is slow as requires me to build every revision that is going to be transported in a bundle.

To avoid conversion when pull occurs between to repos with the extension I want client to be able to tell server "don't convert data to regular format". And bundlecaps seem to be exactly what I want (at least with this patch, and a dozen of lines of code in my extension, I got what I wanted).

> -----Original Message-----
> From: mercurial-devel-bounces@selenic.com [mailto:mercurial-devel-
> bounces@selenic.com] On Behalf Of Wojciech Lopata
> Sent: Friday, August 30, 2013 1:14 PM
> To: mercurial-devel@selenic.com
> Subject: [PATCH] localrepo: send client's bundlecaps to a server during a pull
> 
> # HG changeset patch
> # User Wojciech Lopata <lopek@fb.com>
> # Date 1377891837 25200
> #      Fri Aug 30 12:43:57 2013 -0700
> # Node ID b78c0dcf3d674c23462c1b68dff43e6d3277f295
> # Parent  73d761ed3dbddd06d4a65027ef82bb77ae0ec82f
> localrepo: send client's bundlecaps to a server during a pull
> 
> Thanks to this change extensions that modify format of revlog and are used
> on client and server side will be able to customize format of bundles sent
> during a pull, and possibly make this operation faster.
> 
> diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
> --- a/mercurial/localrepo.py
> +++ b/mercurial/localrepo.py
> @@ -148,6 +148,7 @@
>                                          'dotencode'))
>      openerreqs = set(('revlogv1', 'generaldelta'))
>      requirements = ['revlogv1']
> +    bundlecaps = set()
>      filtername = None
> 
>      def _baserequirements(self, create):
> @@ -1671,7 +1672,7 @@
>                  if remote.capable('getbundle'):
>                      # TODO: get bundlecaps from remote
>                      cg = remote.getbundle('pull', common=common,
> -                                          heads=heads or rheads)
> +                            heads=heads or rheads, bundlecaps=self.bundlecaps)
>                  elif heads is None:
>                      cg = remote.changegroup(fetch, 'pull')
>                  elif not remote.capable('changegroupsubset'):
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@selenic.com
> http://selenic.com/mailman/listinfo/mercurial-devel
Durham Goode - Sept. 4, 2013, 6:15 p.m.
On 8/30/13 1:14 PM, "Wojciech Lopata" <lopek@fb.com> wrote:

>localrepo: send client's bundlecaps to a server during a pull
>
>Thanks to this change extensions that modify format of revlog and are
>used on
>client and server side will be able to customize format of bundles sent
>during
>a pull, and possibly make this operation faster.
>
>diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
>--- a/mercurial/localrepo.py
>+++ b/mercurial/localrepo.py
>@@ -148,6 +148,7 @@
>                                         'dotencode'))
>     openerreqs = set(('revlogv1', 'generaldelta'))
>     requirements = ['revlogv1']
>+    bundlecaps = set()
>     filtername = None
> 
>     def _baserequirements(self, create):
>@@ -1671,7 +1672,7 @@
>                 if remote.capable('getbundle'):
>                     # TODO: get bundlecaps from remote
>                     cg = remote.getbundle('pull', common=common,
>-                                          heads=heads or rheads)
>+                            heads=heads or rheads,
>bundlecaps=self.bundlecaps)
>                 elif heads is None:
>                     cg = remote.changegroup(fetch, 'pull')
>                 elif not remote.capable('changegroupsubset'):

Bundlecaps were only added in May (by Benoit I believe), so it's possible
they just aren't used anywhere yet. I think they aren't so much a property
of the localrepo as they are a result of the intersection of the
localrepo's capabilities and the remoterepo's capabilities.

For example, if localrepo supports tinymanifest and you send the
"tinymanifest" bundlecap, how do you know if the resulting bundle is tiny
or not? The solution is to figure out what bundlecaps work for both ends
and only send those.

So instead of making bundlecaps a property on the repo, perhaps make it a
function that accepts a remoterepo. The function can compare the
remoterepo.capabilities() and determine which bundlecaps work for local
and remote, which can then be passed to remote.getbundle(). This would be
where extensions would override and insert their own bundlecaps if the
remote had the extension as well.

Durham
Matt Mackall - Sept. 5, 2013, 10:38 p.m.
On Fri, 2013-08-30 at 13:14 -0700, Wojciech Lopata wrote:
> # HG changeset patch
> # User Wojciech Lopata <lopek@fb.com>
> # Date 1377891837 25200
> #      Fri Aug 30 12:43:57 2013 -0700
> # Node ID b78c0dcf3d674c23462c1b68dff43e6d3277f295
> # Parent  73d761ed3dbddd06d4a65027ef82bb77ae0ec82f
> localrepo: send client's bundlecaps to a server during a pull
> 
> Thanks to this change extensions that modify format of revlog and are used on
> client and server side will be able to customize format of bundles sent during
> a pull, and possibly make this operation faster.

The rule of client-server negotiation in Mercurial is: server (often
stateless!) reports capabilities, client _makes all decisions_.

This means that the client should not be sending capabilities to the
server, instead it should be saying "I see you support X, give me X."

Patch

diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -148,6 +148,7 @@ 
                                         'dotencode'))
     openerreqs = set(('revlogv1', 'generaldelta'))
     requirements = ['revlogv1']
+    bundlecaps = set()
     filtername = None
 
     def _baserequirements(self, create):
@@ -1671,7 +1672,7 @@ 
                 if remote.capable('getbundle'):
                     # TODO: get bundlecaps from remote
                     cg = remote.getbundle('pull', common=common,
-                                          heads=heads or rheads)
+                            heads=heads or rheads, bundlecaps=self.bundlecaps)
                 elif heads is None:
                     cg = remote.changegroup(fetch, 'pull')
                 elif not remote.capable('changegroupsubset'):