Patchwork D210: pushvars: move fb extension pushvars to core

login
register
mail settings
Submitter phabricator
Date Aug. 2, 2017, 12:08 a.m.
Message ID <differential-rev-PHID-DREV-hs46epeutanre23govge-req@phab.mercurial-scm.org>
Download mbox | patch
Permalink /patch/22627/
State Superseded
Headers show

Comments

phabricator - Aug. 2, 2017, 12:08 a.m.
pulkit created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  pushvars extension in fbext adds a --pushvars flag to push command using which
  one send environment variables to server which can be used to run hooks. The
  extension is moved directly to core with a flag to turn off the unbundling of
  the environment variables on the server which can help security related issues.
  
  This patch also adds the test for the extension.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

AFFECTED FILES
  mercurial/bundle2.py
  mercurial/commands.py
  mercurial/exchange.py
  tests/test-completion.t
  tests/test-pushvars.t

CHANGE DETAILS




To: pulkit, #hg-reviewers
Cc: mercurial-devel
phabricator - Aug. 2, 2017, 1:02 a.m.
akushner added inline comments.

INLINE COMMENTS

> test-pushvars.t:7
> +  $ cat > $TESTTMP/pretxnchangegroup.sh << EOF
> +  > #!/bin/bash
> +  > env | grep -E "^HG_USERVAR_DEBUG"

Please use /bin/sh

> test-pushvars.t:8-9
> +  > #!/bin/bash
> +  > env | grep -E "^HG_USERVAR_DEBUG"
> +  > env | grep -E "^HG_USERVAR_BYPASS_REVIEW"
> +  > exit 0

I didn't see 'grep -E' in the test base, but do see egrep. Consider this instead:

  env |  egrep "^HG_USERVAR_(DEBUG|BYPASS_REVIEW)" | sort

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers
Cc: akushner, mercurial-devel
phabricator - Aug. 2, 2017, 1:05 a.m.
akushner added inline comments.

INLINE COMMENTS

> commands.py:4011
>  
> +    The --pushvars option is used to passed environment variables to server. If
> +    you want to disable this on your server for security purposes, you can add

s/passed/pass/

> commands.py:4012-4016
> +    you want to disable this on your server for security purposes, you can add
> +    the following to your configuration file:
> +
> +    [push]
> +    pushvars.server = no

This should definitely be opt-in, vs. opt-out.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers
Cc: akushner, mercurial-devel
phabricator - Aug. 2, 2017, 1:20 a.m.
pulkit added inline comments.

INLINE COMMENTS

> akushner wrote in commands.py:4012-4016
> This should definitely be opt-in, vs. opt-out.

Sorry but I don't understand what you mean here. Do I need to change the documentation part or the change the flag part. Currently I have set the default of the flag to True.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers
Cc: akushner, mercurial-devel
phabricator - Aug. 2, 2017, 1:28 a.m.
akushner added a comment.


  @pulkit - see what others say. People might disagree with me on this, but I'd rather be cautious on a feature like this. Even though the variables passed are all prepended with HG_USERVAR so I don't see a way how this could cause problems.

INLINE COMMENTS

> pulkit wrote in commands.py:4012-4016
> Sorry but I don't understand what you mean here. Do I need to change the documentation part or the change the flag part. Currently I have set the default of the flag to True.

I think people might have concerns about sending variables to the server so I think this should be default off and have the documentation tell how to enable it.

I would write the documentation such that the meaning would be "if you want to enable this on your server, add the following"

For reference, I created this feature since I used to use a similar feature in git where you'd do the following convolution:

  git push --receive-pack='env ALLOW_CRAZY_FILENAMES=true git-receive-pack' "

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers
Cc: akushner, mercurial-devel
phabricator - Aug. 2, 2017, 3:14 p.m.
durin42 added a comment.


  I've often wanted a way to have extra arguments to push commands (etc), but I'm not sure that just exporting all environment variables is exactly the interface I want. Could we accomplish something similar by having an --extra-args= flag that delivers a part with an arbitrary payload that can then be used (or not) by hooks as they see fit? Or is that too challenging?
  
  (The environment variable model also feels like it fits poorly with http servers, in my brain.)

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers
Cc: durin42, akushner, mercurial-devel
phabricator - Aug. 2, 2017, 4:40 p.m.
akushner added a comment.


  @durin42 - This isn't exporting all the shell variables.... it works almost exactly like you are suggesting
  
  > Could we accomplish something similar by having an --extra-args= flag that delivers a part with an arbitrary payload that can then be used (or not) by hooks as they see fit?
  
  How we use this feature:
  
    hg push --pushvars "BYPASS_LARGE_FILE_CHECK=true"
  
  and then the hook that usually balks when someone tries to upload some ridiculously sized binary does something like
  
    if [[ $HG_USERVAR_BYPASS_LARGE_FILE_CHECK == true ]]; then 
      # Don't bail and allow what we usually don't allow
    fi
  
  Notice that the HG_USERVAR is prepended to the "BYPASS_LARGE_FILE_CHECK" var? The user can't override normal shell variables with this.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers
Cc: durin42, akushner, mercurial-devel
phabricator - Aug. 2, 2017, 5:05 p.m.
durin42 added a comment.


  In https://phab.mercurial-scm.org/D210#3506, @akushner wrote:
  
  > @durin42 - This isn't exporting all the shell variables.... it works almost exactly like you are suggesting
  >
  > > Could we accomplish something similar by having an --extra-args= flag that delivers a part with an arbitrary payload that can then be used (or not) by hooks as they see fit?
  >
  > How we use this feature:
  >
  >   hg push --pushvars "BYPASS_LARGE_FILE_CHECK=true"
  >
  >
  > and then the hook that usually balks when someone tries to upload some ridiculously sized binary does something like
  >
  >   if [[ $HG_USERVAR_BYPASS_LARGE_FILE_CHECK == true ]]; then 
  >     # Don't bail and allow what we usually don't allow
  >   fi
  >
  
  
  Right, that's roughly what I expected. Could it be done using HG_EXTRA_ARGS that was set to whatever --extra-args contained?
  
  > Notice that the HG_USERVAR is prepended to the "BYPASS_LARGE_FILE_CHECK" var? The user can't override normal shell variables with this.
  
  Sure, but environment variables are still global state, so its got nonzero concurrency concerns. I'm also *extremely* uncomfortable shipping *all* environment variables because people put credentials in them on a regular basis, so if we do go with the "ship an environment variable" approach, I think the user should be specifying which variables to send.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers
Cc: durin42, akushner, mercurial-devel
phabricator - Aug. 2, 2017, 5:25 p.m.
pulkit added a comment.


  
  
  >> Notice that the HG_USERVAR is prepended to the "BYPASS_LARGE_FILE_CHECK" var? The user can't override normal shell variables with this.
  > 
  > Sure, but environment variables are still global state, so its got nonzero concurrency concerns. I'm also *extremely* uncomfortable shipping *all* environment variables because people put credentials in them on a regular basis, so if we do go with the "ship an environment variable" approach, I think the user should be specifying which variables to send.
  
  This is not sending all the environment variables of a user. Only the key, values passed with --pushvars are send to server and server has an option whether to unbundle them or not. The keys passed with --pushvars are prepended with HG_USERVAR. I am sorry if the commit description is confusing.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers
Cc: durin42, akushner, mercurial-devel
phabricator - Aug. 2, 2017, 6:51 p.m.
akushner added inline comments.

INLINE COMMENTS

> commands.py:4012
> +    The --pushvars option is used to passed environment variables to server. If
> +    you want to disable this on your server for security purposes, you can add
> +    the following to your configuration file:

The --pushvars option sends strings to the server that become environment variables prepended with HG_USERVAR_. For example, '--pushvars ENABLE_FEATURE=true', provides the server side hooks with 'HG_USERVAR_ENABLE_FEATURE=true' as part of their environment.
  
  Pushvars can provide for user-overridable hooks as well as set debug levels. One example is having a hook that blocks commits containing conflict markers, but enables the user to override the hook if the file is using conflict markers for testing purposes or the file format has strings that look like conflict markers.
  
  To enable this feature on your server, add the following to your configuration file:
  
  [push]
  pushvars.server = true

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers
Cc: durin42, akushner, mercurial-devel
phabricator - Aug. 2, 2017, 7:23 p.m.
quark added inline comments.

INLINE COMMENTS

> commands.py:3973
>      ('', 'new-branch', False, _('allow pushing a new branch')),
> +    ('', 'pushvars', [], _('variables that can be sent to server')),
>      ] + remoteopts,

Maybe add `(ADVANCED)` so it gets hidden in `--help` without `--verbose`.

> akushner wrote in commands.py:4012
>   The --pushvars option sends strings to the server that become environment variables prepended with HG_USERVAR_. For example, '--pushvars ENABLE_FEATURE=true', provides the server side hooks with 'HG_USERVAR_ENABLE_FEATURE=true' as part of their environment.
>   
>   Pushvars can provide for user-overridable hooks as well as set debug levels. One example is having a hook that blocks commits containing conflict markers, but enables the user to override the hook if the file is using conflict markers for testing purposes or the file format has strings that look like conflict markers.
>   
>   To enable this feature on your server, add the following to your configuration file:
>   
>   [push]
>   pushvars.server = true

To better fit the rst format, it might be:

  .. container:: verbose
  
     The --pushvars option sends strings to the server that become environment
     variables prepended with HG_USERVAR_. For example, ``--pushvars
     ENABLE_FEATURE=true`` provides the server side hooks with
     ``HG_USERVAR_ENABLE_FEATURE=true`` as part of their environment.
  
     Pushvars can provide for user-overridable hooks as well as set debug levels.
     One example is having a hook that blocks commits containing conflict
     markers, but enables the user to override the hook if the file is using
     conflict markers for testing purposes or the file format has strings that
     look like conflict markers.
  
     By default, servers will ignore `--pushvars`. To enable it, set
     ``pushvars.server`` set to ``true`` server-side.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers
Cc: quark, durin42, akushner, mercurial-devel
phabricator - Aug. 2, 2017, 8:10 p.m.
quark added inline comments.

INLINE COMMENTS

> quark wrote in commands.py:4012
> To better fit the rst format, it might be:
> 
>   .. container:: verbose
>   
>      The --pushvars option sends strings to the server that become environment
>      variables prepended with HG_USERVAR_. For example, ``--pushvars
>      ENABLE_FEATURE=true`` provides the server side hooks with
>      ``HG_USERVAR_ENABLE_FEATURE=true`` as part of their environment.
>   
>      Pushvars can provide for user-overridable hooks as well as set debug levels.
>      One example is having a hook that blocks commits containing conflict
>      markers, but enables the user to override the hook if the file is using
>      conflict markers for testing purposes or the file format has strings that
>      look like conflict markers.
>   
>      By default, servers will ignore `--pushvars`. To enable it, set
>      ``pushvars.server`` set to ``true`` server-side.

Oops, `set to` should be changed to `to`.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers
Cc: quark, durin42, akushner, mercurial-devel
phabricator - Aug. 4, 2017, 9:50 p.m.
durin42 accepted this revision.
durin42 added a comment.
This revision is now accepted and ready to land.


  I like this, it opens the doors to some things I've wanted to do for a while.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D210

To: pulkit, #hg-reviewers, durin42
Cc: quark, durin42, akushner, mercurial-devel

Patch

diff --git a/tests/test-pushvars.t b/tests/test-pushvars.t
new file mode 100644
--- /dev/null
+++ b/tests/test-pushvars.t
@@ -0,0 +1,58 @@ 
+Setup
+
+  $ PYTHONPATH=$TESTDIR/..:$PYTHONPATH
+  $ export PYTHONPATH
+
+  $ cat > $TESTTMP/pretxnchangegroup.sh << EOF
+  > #!/bin/bash
+  > env | grep -E "^HG_USERVAR_DEBUG"
+  > env | grep -E "^HG_USERVAR_BYPASS_REVIEW"
+  > exit 0
+  > EOF
+  $ chmod +x $TESTTMP/pretxnchangegroup.sh
+  $ cat >> $HGRCPATH << EOF
+  > [hooks]
+  > pretxnchangegroup = $TESTTMP/pretxnchangegroup.sh
+  > [experimental]
+  > bundle2-exp = true
+  > EOF
+
+  $ hg init repo
+  $ hg clone -q repo child
+  $ cd child
+
+Test pushing vars to repo
+
+  $ echo b > a
+  $ hg commit -Aqm a
+  $ hg push --pushvars "DEBUG=1" --pushvars "BYPASS_REVIEW=true"
+  pushing to $TESTTMP/repo (glob)
+  searching for changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  HG_USERVAR_DEBUG=1
+  HG_USERVAR_BYPASS_REVIEW=true
+
+Test pushing var with empty right-hand side
+
+  $ echo b >> a
+  $ hg commit -Aqm a
+  $ hg push --pushvars "DEBUG="
+  pushing to $TESTTMP/repo (glob)
+  searching for changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  HG_USERVAR_DEBUG=
+
+Test pushing bad vars
+
+  $ echo b >> a
+  $ hg commit -Aqm b
+  $ hg push --pushvars "DEBUG"
+  pushing to $TESTTMP/repo (glob)
+  abort: unable to parse variable 'DEBUG', should follow 'KEY=VALUE' or 'KEY=' format
+  [255]
diff --git a/tests/test-completion.t b/tests/test-completion.t
--- a/tests/test-completion.t
+++ b/tests/test-completion.t
@@ -228,7 +228,7 @@ 
   log: follow, follow-first, date, copies, keyword, rev, removed, only-merges, user, only-branch, branch, prune, patch, git, limit, no-merges, stat, graph, style, template, include, exclude
   merge: force, rev, preview, tool
   pull: update, force, rev, bookmark, branch, ssh, remotecmd, insecure
-  push: force, rev, bookmark, branch, new-branch, ssh, remotecmd, insecure
+  push: force, rev, bookmark, branch, new-branch, pushvars, ssh, remotecmd, insecure
   remove: after, force, subrepos, include, exclude
   serve: accesslog, daemon, daemon-postexec, errorlog, port, address, prefix, name, web-conf, webdir-conf, pid-file, stdio, cmdserver, templates, style, ipv6, certificate, subrepos
   status: all, modified, added, removed, deleted, clean, unknown, ignored, no-status, terse, copies, print0, rev, change, include, exclude, subrepos, template
diff --git a/mercurial/exchange.py b/mercurial/exchange.py
--- a/mercurial/exchange.py
+++ b/mercurial/exchange.py
@@ -891,6 +891,14 @@ 
                         pushop.bkresult = 1
     return handlereply
 
+@b2partsgenerator('pushvars', idx=0)
+def _getbundlesendvars(pushop, bundler):
+    '''send shellvars via bundle2'''
+    if getattr(pushop.repo, '_shellvars', ()):
+        part = bundler.newpart('pushvars')
+
+        for key, value in pushop.repo._shellvars.iteritems():
+            part.addparam(key, value, mandatory=False)
 
 def _pushbundle2(pushop):
     """push data to the remote using bundle2
diff --git a/mercurial/commands.py b/mercurial/commands.py
--- a/mercurial/commands.py
+++ b/mercurial/commands.py
@@ -3970,6 +3970,7 @@ 
     ('b', 'branch', [],
      _('a specific branch you would like to push'), _('BRANCH')),
     ('', 'new-branch', False, _('allow pushing a new branch')),
+    ('', 'pushvars', [], _('variables that can be sent to server')),
     ] + remoteopts,
     _('[-f] [-r REV]... [-e CMD] [--remotecmd CMD] [DEST]'))
 def push(ui, repo, dest=None, **opts):
@@ -4007,6 +4008,13 @@ 
     Please see :hg:`help urls` for important details about ``ssh://``
     URLs. If DESTINATION is omitted, a default path will be used.
 
+    The --pushvars option is used to passed environment variables to server. If
+    you want to disable this on your server for security purposes, you can add
+    the following to your configuration file:
+
+    [push]
+    pushvars.server = no
+
     Returns 0 if push was successful, 1 if nothing to push.
     """
 
@@ -4059,11 +4067,28 @@ 
                 return not result
     finally:
         del repo._subtoppath
+
+    pushvars = opts.get('pushvars')
+    if pushvars:
+        shellvars = {}
+        for raw in pushvars:
+            if '=' not in raw:
+                msg = ("unable to parse variable '%s', should follow "
+                        "'KEY=VALUE' or 'KEY=' format")
+                raise error.Abort(msg % raw)
+            k, v = raw.split('=', 1)
+            shellvars[k] = v
+
+        repo._shellvars = shellvars
+
     pushop = exchange.push(repo, other, opts.get('force'), revs=revs,
                            newbranch=opts.get('new_branch'),
                            bookmarks=opts.get('bookmark', ()),
                            opargs=opts.get('opargs'))
 
+    if pushvars:
+        del repo._shellvars
+
     result = not pushop.cgresult
 
     if pushop.bkresult is not None:
diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1872,3 +1872,17 @@ 
 
     cache.write()
     op.ui.debug('applied %i hgtags fnodes cache entries\n' % count)
+
+@parthandler('pushvars')
+def bundle2getvars(op, part):
+    '''unbundle a bundle2 containing shellvars on the server'''
+    # An option to disable unbundling on server-side for security reasons
+    if op.ui.configbool('push', 'pushvars.server', True):
+        hookargs = {}
+        for key, value in part.advisoryparams:
+            key = key.upper()
+            # We want pushed variables to have USERVAR_ prepended so we know
+            # they came from the --pushvar flag.
+            key = "USERVAR_" + key
+            hookargs[key] = value
+        op.addhookargs(hookargs)