Patchwork [5,of,5,RFC] exchange: support phases discovery via public heads

login
register
mail settings
Submitter Gregory Szorc
Date Feb. 16, 2016, 2:03 a.m.
Message ID <d51ed0ddd05c3c716df5.1455588216@ubuntu-vm-main>
Download mbox | patch
Permalink /patch/13222/
State Accepted
Headers show

Comments

Gregory Szorc - Feb. 16, 2016, 2:03 a.m.
# HG changeset patch
# User Gregory Szorc <gregory.szorc@gmail.com>
# Date 1455588068 28800
#      Mon Feb 15 18:01:08 2016 -0800
# Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c
# Parent  8b939990c70a4e545f18db3338f7798f09a94ada
exchange: support phases discovery via public heads

Now that we can detect when the server supports exposing phases via
public heads, we leverage this namespace as part of phases discovery.

On Mozilla's Try repository, the phases listkeys response drops from
~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes.
Mike Hommey - Feb. 16, 2016, 2:47 a.m.
On Mon, Feb 15, 2016 at 06:03:36PM -0800, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc@gmail.com>
> # Date 1455588068 28800
> #      Mon Feb 15 18:01:08 2016 -0800
> # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c
> # Parent  8b939990c70a4e545f18db3338f7798f09a94ada
> exchange: support phases discovery via public heads
> 
> Now that we can detect when the server supports exposing phases via
> public heads, we leverage this namespace as part of phases discovery.
> 
> On Mozilla's Try repository, the phases listkeys response drops from
> ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes.

While this works with Mozilla's Try, what about hypothetical(?)
heavy-headed *publishing* repositories? This set of changes would make
them go the opposite direction.

Mike
Gregory Szorc - Feb. 16, 2016, 3:15 a.m.
On Mon, Feb 15, 2016 at 6:47 PM, Mike Hommey <mh@glandium.org> wrote:

> On Mon, Feb 15, 2016 at 06:03:36PM -0800, Gregory Szorc wrote:
> > # HG changeset patch
> > # User Gregory Szorc <gregory.szorc@gmail.com>
> > # Date 1455588068 28800
> > #      Mon Feb 15 18:01:08 2016 -0800
> > # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c
> > # Parent  8b939990c70a4e545f18db3338f7798f09a94ada
> > exchange: support phases discovery via public heads
> >
> > Now that we can detect when the server supports exposing phases via
> > public heads, we leverage this namespace as part of phases discovery.
> >
> > On Mozilla's Try repository, the phases listkeys response drops from
> > ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes.
>
> While this works with Mozilla's Try, what about hypothetical(?)
> heavy-headed *publishing* repositories? This set of changes would make
> them go the opposite direction.
>

Yup. Hence the RFC. We probably want a server-side config knob or something
on the server that tells the client whether to prefer draft roots or public
heads depending on which has fewer entries.
Pierre-Yves David - Feb. 20, 2016, 5:20 p.m.
On 02/16/2016 03:03 AM, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc@gmail.com>
> # Date 1455588068 28800
> #      Mon Feb 15 18:01:08 2016 -0800
> # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c
> # Parent  8b939990c70a4e545f18db3338f7798f09a94ada
> exchange: support phases discovery via public heads
>
> Now that we can detect when the server supports exposing phases via
> public heads, we leverage this namespace as part of phases discovery.
>
> On Mozilla's Try repository, the phases listkeys response drops from
> ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes.

I think the discovery process need an overall rework, There is multiple 
types of data running there own discovery with a lot of overlap and race 
condition.

I would rather go in that direction instead. What do you think?
Gregory Szorc - Feb. 21, 2016, 12:12 a.m.
On Sat, Feb 20, 2016 at 9:20 AM, Pierre-Yves David <
pierre-yves.david@ens-lyon.org> wrote:

>
>
> On 02/16/2016 03:03 AM, Gregory Szorc wrote:
>
>> # HG changeset patch
>> # User Gregory Szorc <gregory.szorc@gmail.com>
>> # Date 1455588068 28800
>> #      Mon Feb 15 18:01:08 2016 -0800
>> # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c
>> # Parent  8b939990c70a4e545f18db3338f7798f09a94ada
>> exchange: support phases discovery via public heads
>>
>> Now that we can detect when the server supports exposing phases via
>> public heads, we leverage this namespace as part of phases discovery.
>>
>> On Mozilla's Try repository, the phases listkeys response drops from
>> ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes.
>>
>
> I think the discovery process need an overall rework, There is multiple
> types of data running there own discovery with a lot of overlap and race
> condition.
>
> I would rather go in that direction instead. What do you think?


I agree that discovery needs an overhaul. I'm scared about how much work
that will be. I was kinda hoping to make incremental improvements to phases
(this series) and improve discovery in a few key scenarios (such as a force
push of a single head to a mega-headed repo and perhaps a mode that does
discovery without taking non-public changesets into consideration).

Since you are the discovery guru, would you mind starting a *Plan wiki page
documenting the existing discovery deficiencies so I can start
brainstorming?
Pierre-Yves David - Feb. 21, 2016, 9:22 a.m.
On 02/21/2016 01:12 AM, Gregory Szorc wrote:
> On Sat, Feb 20, 2016 at 9:20 AM, Pierre-Yves David
> <pierre-yves.david@ens-lyon.org <mailto:pierre-yves.david@ens-lyon.org>>
> wrote:
>
>
>
>     On 02/16/2016 03:03 AM, Gregory Szorc wrote:
>
>         # HG changeset patch
>         # User Gregory Szorc <gregory.szorc@gmail.com
>         <mailto:gregory.szorc@gmail.com>>
>         # Date 1455588068 28800
>         #      Mon Feb 15 18:01:08 2016 -0800
>         # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c
>         # Parent  8b939990c70a4e545f18db3338f7798f09a94ada
>         exchange: support phases discovery via public heads
>
>         Now that we can detect when the server supports exposing phases via
>         public heads, we leverage this namespace as part of phases
>         discovery.
>
>         On Mozilla's Try repository, the phases listkeys response drops from
>         ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes.
>
>
>     I think the discovery process need an overall rework, There is
>     multiple types of data running there own discovery with a lot of
>     overlap and race condition.
>
>     I would rather go in that direction instead. What do you think?
>
>
> I agree that discovery needs an overhaul. I'm scared about how much work
> that will be. I was kinda hoping to make incremental improvements to
> phases (this series) and improve discovery in a few key scenarios (such
> as a force push of a single head to a mega-headed repo and perhaps a
> mode that does discovery without taking non-public changesets into
> consideration).

I think we can make small steps here. Changing the protocol to be 
extensible (sending multiple types of data per roundtrip) and then we 
can slowly extend the type of data we need.

> Since you are the discovery guru, would you mind starting a *Plan wiki
> page documenting the existing discovery deficiencies so I can start
> brainstorming?

Good idea, I'll try to get that rolling soon. Thanks for the nudge.
Martin von Zweigbergk - Feb. 21, 2016, 4:16 p.m.
The updates to the discovery protocol sound interesting even though I don't
know much about it. Worth taking about at the sprint? Or perhaps it will be
pretty clear after the Plan page is created...

On Sun, Feb 21, 2016, 01:22 Pierre-Yves David <
pierre-yves.david@ens-lyon.org> wrote:

>
>
> On 02/21/2016 01:12 AM, Gregory Szorc wrote:
> > On Sat, Feb 20, 2016 at 9:20 AM, Pierre-Yves David
> > <pierre-yves.david@ens-lyon.org <mailto:pierre-yves.david@ens-lyon.org>>
> > wrote:
> >
> >
> >
> >     On 02/16/2016 03:03 AM, Gregory Szorc wrote:
> >
> >         # HG changeset patch
> >         # User Gregory Szorc <gregory.szorc@gmail.com
> >         <mailto:gregory.szorc@gmail.com>>
> >         # Date 1455588068 28800
> >         #      Mon Feb 15 18:01:08 2016 -0800
> >         # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c
> >         # Parent  8b939990c70a4e545f18db3338f7798f09a94ada
> >         exchange: support phases discovery via public heads
> >
> >         Now that we can detect when the server supports exposing phases
> via
> >         public heads, we leverage this namespace as part of phases
> >         discovery.
> >
> >         On Mozilla's Try repository, the phases listkeys response drops
> from
> >         ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes.
> >
> >
> >     I think the discovery process need an overall rework, There is
> >     multiple types of data running there own discovery with a lot of
> >     overlap and race condition.
> >
> >     I would rather go in that direction instead. What do you think?
> >
> >
> > I agree that discovery needs an overhaul. I'm scared about how much work
> > that will be. I was kinda hoping to make incremental improvements to
> > phases (this series) and improve discovery in a few key scenarios (such
> > as a force push of a single head to a mega-headed repo and perhaps a
> > mode that does discovery without taking non-public changesets into
> > consideration).
>
> I think we can make small steps here. Changing the protocol to be
> extensible (sending multiple types of data per roundtrip) and then we
> can slowly extend the type of data we need.
>
> > Since you are the discovery guru, would you mind starting a *Plan wiki
> > page documenting the existing discovery deficiencies so I can start
> > brainstorming?
>
> Good idea, I'll try to get that rolling soon. Thanks for the nudge.
>
> --
> Pierre-Yves David
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>
Pierre-Yves David - Feb. 21, 2016, 5:48 p.m.
On 02/21/2016 05:16 PM, Martin von Zweigbergk wrote:
> The updates to the discovery protocol sound interesting even though I
> don't know much about it. Worth taking about at the sprint? Or perhaps
> it will be pretty clear after the Plan page is created...

I've boostrapped a page, have fun

https://www.mercurial-scm.org/wiki/ExtensibleDiscoveryPlan

Patch

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -8,16 +8,17 @@ 
 from __future__ import absolute_import
 
 import errno
 import urllib
 import urllib2
 
 from .i18n import _
 from .node import (
+    bin,
     hex,
     nullid,
 )
 from . import (
     base85,
     bookmarks as bookmod,
     bundle2,
     changegroup,
@@ -506,36 +507,45 @@  def _pushdiscoverychangeset(pushop):
 
 @pushdiscovery('phase')
 def _pushdiscoveryphase(pushop):
     """discover the phase that needs to be pushed
 
     (computed for both success and failure case for changesets push)"""
     outgoing = pushop.outgoing
     unfi = pushop.repo.unfiltered()
-    remotephases = pushop.remote.listkeys('phases')
-    publishing = remotephases.get('publishing', False)
-    if (pushop.ui.configbool('ui', '_usedassubrepo', False)
-        and remotephases    # server supports phases
-        and not pushop.outgoing.missing # no changesets to be pushed
-        and publishing):
-        # When:
-        # - this is a subrepo push
-        # - and remote support phase
-        # - and no changeset are to be pushed
-        # - and remote is publishing
-        # We may be in issue 3871 case!
-        # We drop the possible phase synchronisation done by
-        # courtesy to publish changesets possibly locally draft
-        # on the remote.
-        remotephases = {'publishing': 'True'}
-    ana = phases.analyzeremotephases(pushop.repo,
-                                     pushop.fallbackheads,
-                                     remotephases)
-    pheads, droots = ana
+
+    if pushop.remote.capable('nonpublishing'):
+        publishing = False
+        publicheads = pushop.remote.listkeys('publicphases')
+        pheads = [bin(n) for n in publicheads]
+        revset = pushop.repo.set('roots(children(%ln))', pheads)
+        droots = [c.node() for c in revset]
+    else:
+        remotephases = pushop.remote.listkeys('phases')
+        publishing = remotephases.get('publishing', False)
+        if (pushop.ui.configbool('ui', '_usedassubrepo', False)
+            and remotephases    # server supports phases
+            and not pushop.outgoing.missing # no changesets to be pushed
+            and publishing):
+            # When:
+            # - this is a subrepo push
+            # - and remote support phase
+            # - and no changeset are to be pushed
+            # - and remote is publishing
+            # We may be in issue 3871 case!
+            # We drop the possible phase synchronisation done by
+            # courtesy to publish changesets possibly locally draft
+            # on the remote.
+            remotephases = {'publishing': 'True'}
+        ana = phases.analyzeremotephases(pushop.repo,
+                                         pushop.fallbackheads,
+                                         remotephases)
+        pheads, droots = ana
+
     extracond = ''
     if not publishing:
         extracond = ' and public()'
     revset = 'heads((%%ln::%%ln) %s)' % extracond
     # Get the list of all revs draft on remote by public here.
     # XXX Beware that revset break if droots is not strictly
     # XXX root we may want to ensure it is but it is costly
     fallback = list(unfi.set(revset, droots, pushop.fallbackheads))