| Submitter | Gregory Szorc |
|---|---|
| Date | Feb. 16, 2016, 2:03 a.m. |
| Message ID | <d51ed0ddd05c3c716df5.1455588216@ubuntu-vm-main> |
| Download | mbox | patch |
| Permalink | /patch/13222/ |
| State | Accepted |
| Headers | show |
Comments
On Mon, Feb 15, 2016 at 06:03:36PM -0800, Gregory Szorc wrote: > # HG changeset patch > # User Gregory Szorc <gregory.szorc@gmail.com> > # Date 1455588068 28800 > # Mon Feb 15 18:01:08 2016 -0800 > # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c > # Parent 8b939990c70a4e545f18db3338f7798f09a94ada > exchange: support phases discovery via public heads > > Now that we can detect when the server supports exposing phases via > public heads, we leverage this namespace as part of phases discovery. > > On Mozilla's Try repository, the phases listkeys response drops from > ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes. While this works with Mozilla's Try, what about hypothetical(?) heavy-headed *publishing* repositories? This set of changes would make them go the opposite direction. Mike
On Mon, Feb 15, 2016 at 6:47 PM, Mike Hommey <mh@glandium.org> wrote: > On Mon, Feb 15, 2016 at 06:03:36PM -0800, Gregory Szorc wrote: > > # HG changeset patch > > # User Gregory Szorc <gregory.szorc@gmail.com> > > # Date 1455588068 28800 > > # Mon Feb 15 18:01:08 2016 -0800 > > # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c > > # Parent 8b939990c70a4e545f18db3338f7798f09a94ada > > exchange: support phases discovery via public heads > > > > Now that we can detect when the server supports exposing phases via > > public heads, we leverage this namespace as part of phases discovery. > > > > On Mozilla's Try repository, the phases listkeys response drops from > > ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes. > > While this works with Mozilla's Try, what about hypothetical(?) > heavy-headed *publishing* repositories? This set of changes would make > them go the opposite direction. > Yup. Hence the RFC. We probably want a server-side config knob or something on the server that tells the client whether to prefer draft roots or public heads depending on which has fewer entries.
On 02/16/2016 03:03 AM, Gregory Szorc wrote: > # HG changeset patch > # User Gregory Szorc <gregory.szorc@gmail.com> > # Date 1455588068 28800 > # Mon Feb 15 18:01:08 2016 -0800 > # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c > # Parent 8b939990c70a4e545f18db3338f7798f09a94ada > exchange: support phases discovery via public heads > > Now that we can detect when the server supports exposing phases via > public heads, we leverage this namespace as part of phases discovery. > > On Mozilla's Try repository, the phases listkeys response drops from > ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes. I think the discovery process need an overall rework, There is multiple types of data running there own discovery with a lot of overlap and race condition. I would rather go in that direction instead. What do you think?
On Sat, Feb 20, 2016 at 9:20 AM, Pierre-Yves David < pierre-yves.david@ens-lyon.org> wrote: > > > On 02/16/2016 03:03 AM, Gregory Szorc wrote: > >> # HG changeset patch >> # User Gregory Szorc <gregory.szorc@gmail.com> >> # Date 1455588068 28800 >> # Mon Feb 15 18:01:08 2016 -0800 >> # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c >> # Parent 8b939990c70a4e545f18db3338f7798f09a94ada >> exchange: support phases discovery via public heads >> >> Now that we can detect when the server supports exposing phases via >> public heads, we leverage this namespace as part of phases discovery. >> >> On Mozilla's Try repository, the phases listkeys response drops from >> ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes. >> > > I think the discovery process need an overall rework, There is multiple > types of data running there own discovery with a lot of overlap and race > condition. > > I would rather go in that direction instead. What do you think? I agree that discovery needs an overhaul. I'm scared about how much work that will be. I was kinda hoping to make incremental improvements to phases (this series) and improve discovery in a few key scenarios (such as a force push of a single head to a mega-headed repo and perhaps a mode that does discovery without taking non-public changesets into consideration). Since you are the discovery guru, would you mind starting a *Plan wiki page documenting the existing discovery deficiencies so I can start brainstorming?
On 02/21/2016 01:12 AM, Gregory Szorc wrote: > On Sat, Feb 20, 2016 at 9:20 AM, Pierre-Yves David > <pierre-yves.david@ens-lyon.org <mailto:pierre-yves.david@ens-lyon.org>> > wrote: > > > > On 02/16/2016 03:03 AM, Gregory Szorc wrote: > > # HG changeset patch > # User Gregory Szorc <gregory.szorc@gmail.com > <mailto:gregory.szorc@gmail.com>> > # Date 1455588068 28800 > # Mon Feb 15 18:01:08 2016 -0800 > # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c > # Parent 8b939990c70a4e545f18db3338f7798f09a94ada > exchange: support phases discovery via public heads > > Now that we can detect when the server supports exposing phases via > public heads, we leverage this namespace as part of phases > discovery. > > On Mozilla's Try repository, the phases listkeys response drops from > ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes. > > > I think the discovery process need an overall rework, There is > multiple types of data running there own discovery with a lot of > overlap and race condition. > > I would rather go in that direction instead. What do you think? > > > I agree that discovery needs an overhaul. I'm scared about how much work > that will be. I was kinda hoping to make incremental improvements to > phases (this series) and improve discovery in a few key scenarios (such > as a force push of a single head to a mega-headed repo and perhaps a > mode that does discovery without taking non-public changesets into > consideration). I think we can make small steps here. Changing the protocol to be extensible (sending multiple types of data per roundtrip) and then we can slowly extend the type of data we need. > Since you are the discovery guru, would you mind starting a *Plan wiki > page documenting the existing discovery deficiencies so I can start > brainstorming? Good idea, I'll try to get that rolling soon. Thanks for the nudge.
The updates to the discovery protocol sound interesting even though I don't know much about it. Worth taking about at the sprint? Or perhaps it will be pretty clear after the Plan page is created... On Sun, Feb 21, 2016, 01:22 Pierre-Yves David < pierre-yves.david@ens-lyon.org> wrote: > > > On 02/21/2016 01:12 AM, Gregory Szorc wrote: > > On Sat, Feb 20, 2016 at 9:20 AM, Pierre-Yves David > > <pierre-yves.david@ens-lyon.org <mailto:pierre-yves.david@ens-lyon.org>> > > wrote: > > > > > > > > On 02/16/2016 03:03 AM, Gregory Szorc wrote: > > > > # HG changeset patch > > # User Gregory Szorc <gregory.szorc@gmail.com > > <mailto:gregory.szorc@gmail.com>> > > # Date 1455588068 28800 > > # Mon Feb 15 18:01:08 2016 -0800 > > # Node ID d51ed0ddd05c3c716df5fbd2e811285bcac7366c > > # Parent 8b939990c70a4e545f18db3338f7798f09a94ada > > exchange: support phases discovery via public heads > > > > Now that we can detect when the server supports exposing phases > via > > public heads, we leverage this namespace as part of phases > > discovery. > > > > On Mozilla's Try repository, the phases listkeys response drops > from > > ~64k entries / 2.8 MB to < 1000 entries and < 65 kbytes. > > > > > > I think the discovery process need an overall rework, There is > > multiple types of data running there own discovery with a lot of > > overlap and race condition. > > > > I would rather go in that direction instead. What do you think? > > > > > > I agree that discovery needs an overhaul. I'm scared about how much work > > that will be. I was kinda hoping to make incremental improvements to > > phases (this series) and improve discovery in a few key scenarios (such > > as a force push of a single head to a mega-headed repo and perhaps a > > mode that does discovery without taking non-public changesets into > > consideration). > > I think we can make small steps here. Changing the protocol to be > extensible (sending multiple types of data per roundtrip) and then we > can slowly extend the type of data we need. > > > Since you are the discovery guru, would you mind starting a *Plan wiki > > page documenting the existing discovery deficiencies so I can start > > brainstorming? > > Good idea, I'll try to get that rolling soon. Thanks for the nudge. > > -- > Pierre-Yves David > _______________________________________________ > Mercurial-devel mailing list > Mercurial-devel@mercurial-scm.org > https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel >
On 02/21/2016 05:16 PM, Martin von Zweigbergk wrote: > The updates to the discovery protocol sound interesting even though I > don't know much about it. Worth taking about at the sprint? Or perhaps > it will be pretty clear after the Plan page is created... I've boostrapped a page, have fun https://www.mercurial-scm.org/wiki/ExtensibleDiscoveryPlan
Patch
diff --git a/mercurial/exchange.py b/mercurial/exchange.py --- a/mercurial/exchange.py +++ b/mercurial/exchange.py @@ -8,16 +8,17 @@ from __future__ import absolute_import import errno import urllib import urllib2 from .i18n import _ from .node import ( + bin, hex, nullid, ) from . import ( base85, bookmarks as bookmod, bundle2, changegroup, @@ -506,36 +507,45 @@ def _pushdiscoverychangeset(pushop): @pushdiscovery('phase') def _pushdiscoveryphase(pushop): """discover the phase that needs to be pushed (computed for both success and failure case for changesets push)""" outgoing = pushop.outgoing unfi = pushop.repo.unfiltered() - remotephases = pushop.remote.listkeys('phases') - publishing = remotephases.get('publishing', False) - if (pushop.ui.configbool('ui', '_usedassubrepo', False) - and remotephases # server supports phases - and not pushop.outgoing.missing # no changesets to be pushed - and publishing): - # When: - # - this is a subrepo push - # - and remote support phase - # - and no changeset are to be pushed - # - and remote is publishing - # We may be in issue 3871 case! - # We drop the possible phase synchronisation done by - # courtesy to publish changesets possibly locally draft - # on the remote. - remotephases = {'publishing': 'True'} - ana = phases.analyzeremotephases(pushop.repo, - pushop.fallbackheads, - remotephases) - pheads, droots = ana + + if pushop.remote.capable('nonpublishing'): + publishing = False + publicheads = pushop.remote.listkeys('publicphases') + pheads = [bin(n) for n in publicheads] + revset = pushop.repo.set('roots(children(%ln))', pheads) + droots = [c.node() for c in revset] + else: + remotephases = pushop.remote.listkeys('phases') + publishing = remotephases.get('publishing', False) + if (pushop.ui.configbool('ui', '_usedassubrepo', False) + and remotephases # server supports phases + and not pushop.outgoing.missing # no changesets to be pushed + and publishing): + # When: + # - this is a subrepo push + # - and remote support phase + # - and no changeset are to be pushed + # - and remote is publishing + # We may be in issue 3871 case! + # We drop the possible phase synchronisation done by + # courtesy to publish changesets possibly locally draft + # on the remote. + remotephases = {'publishing': 'True'} + ana = phases.analyzeremotephases(pushop.repo, + pushop.fallbackheads, + remotephases) + pheads, droots = ana + extracond = '' if not publishing: extracond = ' and public()' revset = 'heads((%%ln::%%ln) %s)' % extracond # Get the list of all revs draft on remote by public here. # XXX Beware that revset break if droots is not strictly # XXX root we may want to ensure it is but it is costly fallback = list(unfi.set(revset, droots, pushop.fallbackheads))