Patchwork D6218: narrow: send specs as bundle2 data instead of param (issue5952) (issue6019)

login
register
mail settings
Submitter phabricator
Date April 8, 2019, 3:23 p.m.
Message ID <differential-rev-PHID-DREV-kikc66knuz4r6424mji6-req@phab.mercurial-scm.org>
Download mbox | patch
Permalink /patch/39535/
State Superseded
Headers show

Comments

phabricator - April 8, 2019, 3:23 p.m.
pulkit created this revision.
Herald added a reviewer: durin42.
Herald added a reviewer: martinvonz.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Before this patch, when ACL is involved, narrowspecs are send as bundle2
  parameter for narrow:spec bundle2 part. The limitation of bundle2 parts are they
  cannot send data larger than 255 bytes. Includes and excludes in narrow are not
  limited by size and they can grow over 255 bytes.
  
  This patch start sending them as bundle2 data. After this change, we try to read
  specs both from parameters and data, making it compatible with older servers.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

AFFECTED FILES
  hgext/narrow/narrowbundle2.py
  mercurial/exchange.py

CHANGE DETAILS




To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: mercurial-devel
phabricator - April 8, 2019, 3:25 p.m.
pulkit added inline comments.

INLINE COMMENTS

> exchange.py:2217
>          narrowspecpart = bundler.newpart('narrow:spec')
> +        data = ''
>          if include:

I don't feel good about the fact that we are not encoding data here. Is there exists some function which I can use to encode and decode this list of specs?

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: mercurial-devel
phabricator - April 8, 2019, 3:48 p.m.
martinvonz added inline comments.

INLINE COMMENTS

> pulkit wrote in exchange.py:2217
> I don't feel good about the fact that we are not encoding data here. Is there exists some function which I can use to encode and decode this list of specs?

You could probably reuse the function from https://phab.mercurial-scm.org/D6184

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: mercurial-devel
phabricator - April 16, 2019, 4:03 p.m.
martinvonz added inline comments.

INLINE COMMENTS

> narrowbundle2.py:152
> +        inc, exc = data.split('\0')
> +        if inc:
> +            includepats |= set(inc.splitlines())

I'd drop these checks

> exchange.py:2218
> +        data = ''
>          if include:
> +            data += '\n'.join(include)

And these

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: idlsoft, mercurial-devel
phabricator - April 16, 2019, 5:11 p.m.
pulkit added inline comments.

INLINE COMMENTS

> martinvonz wrote in narrowbundle2.py:152
> I'd drop these checks

Sent https://phab.mercurial-scm.org/D6241.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: idlsoft, mercurial-devel
phabricator - April 17, 2019, 10:04 a.m.
indygreg added a comment.


  This patch is backwards incompatible over the wire protocol.
  
  The problem is a new client will blindly send part data to an old server expecting part parameters. The old server won't read the part data and it would be as if the includes and excludes were not sent.
  
  We need some kind of capability negotiation that allows the client to opt in to the newer behavior if the server advertises support for it.
  
  Also, my personal preference is to create new bundle2 parts rather than change behavior of existing bundle2 parts. Doing things this way ensures that behavior for a named bundle2 part is constant over time. This keeps implementations simpler, as individual part handling can do one thing and one thing only.
  
  Finally, the internals help docs should be updated to reflect changes to bundle2 part behavior.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: indygreg, idlsoft, mercurial-devel
phabricator - April 17, 2019, 3:08 p.m.
martinvonz added a comment.


  In https://phab.mercurial-scm.org/D6218#91058, @indygreg wrote:
  
  > This patch is backwards incompatible over the wire protocol.
  >
  > The problem is a new client will blindly send part data to an old server expecting part parameters. The old server won't read the part data and it would be as if the includes and excludes were not sent.
  
  
  It's an experimental feature and I suspect it's used only by Sandu (@idlsoft). Sandu, if we released this without the capability negotiation that Greg is talking about, you would need to make sure to upgrade the server before you upgrade your client(s). Are you okay with that? Is anyone aware of any other users of this feature? Greg, are you okay with making a breaking change (to an experimental feature) if the few existing users are okay with it?
  
  > We need some kind of capability negotiation that allows the client to opt in to the newer behavior if the server advertises support for it.
  > 
  > Also, my personal preference is to create new bundle2 parts rather than change behavior of existing bundle2 parts. Doing things this way ensures that behavior for a named bundle2 part is constant over time. This keeps implementations simpler, as individual part handling can do one thing and one thing only.
  > 
  > Finally, the internals help docs should be updated to reflect changes to bundle2 part behavior.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: indygreg, idlsoft, mercurial-devel
phabricator - April 17, 2019, 3:22 p.m.
pulkit added a comment.


  In https://phab.mercurial-scm.org/D6218#91119, @martinvonz wrote:
  
  > In https://phab.mercurial-scm.org/D6218#91058, @indygreg wrote:
  >
  > > This patch is backwards incompatible over the wire protocol.
  > >
  > > The problem is a new client will blindly send part data to an old server expecting part parameters. The old server won't read the part data and it would be as if the includes and excludes were not sent.
  >
  >
  > It's an experimental feature and I suspect it's used only by Sandu (@idlsoft). Sandu, if we released this without the capability negotiation that Greg is talking about, you would need to make sure to upgrade the server before you upgrade your client(s). Are you okay with that? Is anyone aware of any other users of this feature? Greg, are you okay with making a breaking change (to an experimental feature) if the few existing users are okay with it?
  
  
  I agree with @martinvonz here. narrow extension is experimental right now, in 4.9 we had a lot of breaking changes. The narrowspecs are only send back in case when ACL is enabled. If there are users who rely on existing behavior, they must have hit the bug just like @idlsoft  hit.
  
  I am not sure how we can keep sending narrowspecs back using bundle2 param and fix the issues which this patch is trying to.
  
  > 
  > 
  >> We need some kind of capability negotiation that allows the client to opt in to the newer behavior if the server advertises support for it.
  >> 
  >> Also, my personal preference is to create new bundle2 parts rather than change behavior of existing bundle2 parts. Doing things this way ensures that behavior for a named bundle2 part is constant over time. This keeps implementations simpler, as individual part handling can do one thing and one thing only.
  >> 
  >> Finally, the internals help docs should be updated to reflect changes to bundle2 part behavior.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: indygreg, idlsoft, mercurial-devel
phabricator - April 17, 2019, 4:37 p.m.
idlsoft added a comment.


  Because the current client ignores the data completely, the only way to force it to fail I think is to change the name of the part.
  This would make things cleaner probably, but I'll deal with whatever solution you guys settle on.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: indygreg, idlsoft, mercurial-devel
phabricator - April 17, 2019, 4:44 p.m.
martinvonz added a comment.


  In https://phab.mercurial-scm.org/D6218#91121, @idlsoft wrote:
  
  > Because the current client ignores the data completely, the only way to force it to fail I think is to change the name of the part.
  >  This would make things cleaner probably, but I'll deal with whatever solution you guys settle on.
  
  
  I think bundle2 parts can be marked mandatory (by using uppercase in their name?). It seems to me like the ACL part should be mandatory. Is that correct, Sandu? So that's a good point and thanks for mentioning that. Pulkit, I think it's enough to change the name to be something like `narrow:Spec` or `Narrow:spec` (I'm thinking the former since some narrow parts are mandatory and some are not and then they all still start with `narrow:`).

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: indygreg, idlsoft, mercurial-devel
phabricator - April 17, 2019, 5:09 p.m.
idlsoft added a comment.


  If ACL is enabled, processing this part is mandatory, yes.
  On clone, or pull the user doesn't specify includes, so reading this part is the only way the client can get them.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: indygreg, idlsoft, mercurial-devel
phabricator - April 18, 2019, 2:43 p.m.
idlsoft added a comment.


  This is nitpicking, but there is a duplicate `_NARROWACL_SECTION` definition in narrowbundle2.py,
  I think only the one in exchange.py should remain.
  Btw it's still 'narrowhgacl' from the old days.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: indygreg, idlsoft, mercurial-devel
phabricator - April 24, 2019, 6:40 p.m.
pulkit added a comment.


  I created a new version at https://phab.mercurial-scm.org/D6310.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6218

To: pulkit, durin42, martinvonz, #hg-reviewers
Cc: indygreg, idlsoft, mercurial-devel

Patch

diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -2214,12 +2214,14 @@ 
     if (kwargs.get(r'narrow', False) and kwargs.get(r'narrow_acl', False)
         and (include or exclude)):
         narrowspecpart = bundler.newpart('narrow:spec')
+        data = ''
         if include:
-            narrowspecpart.addparam(
-                'include', '\n'.join(include), mandatory=True)
+            data += '\n'.join(include)
+        data += '\0'
         if exclude:
-            narrowspecpart.addparam(
-                'exclude', '\n'.join(exclude), mandatory=True)
+            data += '\n'.join(exclude)
+
+        narrowspecpart.data = data
 
 @getbundle2partsgenerator('bookmarks')
 def _getbundlebookmarkpart(bundler, repo, source, bundlecaps=None,
diff --git a/hgext/narrow/narrowbundle2.py b/hgext/narrow/narrowbundle2.py
--- a/hgext/narrow/narrowbundle2.py
+++ b/hgext/narrow/narrowbundle2.py
@@ -144,6 +144,12 @@ 
 def _handlechangespec_2(op, inpart):
     includepats = set(inpart.params.get(_SPECPART_INCLUDE, '').splitlines())
     excludepats = set(inpart.params.get(_SPECPART_EXCLUDE, '').splitlines())
+    data = inpart.read()
+    inc, exc = data.split('\0')
+    if inc:
+        includepats |= set(inc.splitlines())
+    if exc:
+        excludepats |= set(exc.splitlines())
     narrowspec.validatepatterns(includepats)
     narrowspec.validatepatterns(excludepats)