Patchwork [1,of,2] windows: add a method to convert Unix style command lines to Windows style

login
register
mail settings
Submitter Matt Harbison
Date June 27, 2018, 12:44 p.m.
Message ID <7ac9de5a8826fc95864e.1530103466@Envy>
Download mbox | patch
Permalink /patch/32458/
State Accepted
Headers show

Comments

Matt Harbison - June 27, 2018, 12:44 p.m.
# HG changeset patch
# User Matt Harbison <matt_harbison@yahoo.com>
# Date 1529817189 14400
#      Sun Jun 24 01:13:09 2018 -0400
# Node ID 7ac9de5a8826fc95864ee4ba844eb8b5c9e71332
# Parent  2c2e82469b8915c8153979cd89a970b7317f882d
windows: add a method to convert Unix style command lines to Windows style

This started as a copy/paste of `os.path.expandvars()`, but limited to a given
dictionary of variables, converting `foo = foo + bar` to `foo += bar`, and
adding 'b' string prefixes.  Then code was added to make sure that a value being
substituted in wouldn't itself be expanded by cmd.exe.  But that left
inconsistent results between `$var1` and `%var1%` when its value was '%foo%'-
since neither were touched, `$var1` wouldn't expand but `%var1%` would.  So
instead, this just converts the Unix style to Windows style (if the variable
exists, because Windows will leave `%missing%` as-is), and lets cmd.exe do its
thing.

I then dropped the %% -> % conversion (because Windows doesn't do this), and
added the ability to escape the '$' with '\'.  The escape character is dropped,
for consistency with shell handling.

After everything seemed stable and working, running the whole test suite flagged
a problem near the end of test-bookmarks.t:1069.  The problem is cmd.exe won't
pass empty variables to its child, so defined but empty variables are now
skipped.  I can't think of anything better, and it seems like a pre-existing
violation of the documentation, which calls out that HG_OLDNODE is empty on
bookmark creation.

Future additions could potentially be replacing strong quotes with double quotes
(cmd.exe doesn't know what to do with the former), escaping a double quote, and
some tilde expansion via os.path.expanduser().  I've got some doubts about
replacing the strong quotes in case sh.exe is run, but it seems like the right
thing to do the vast majority of the time.  The original form of this was
discussed about a year ago[1].

[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-July/100735.html
Yuya Nishihara - June 28, 2018, 11:49 a.m.
On Wed, 27 Jun 2018 08:44:26 -0400, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison <matt_harbison@yahoo.com>
> # Date 1529817189 14400
> #      Sun Jun 24 01:13:09 2018 -0400
> # Node ID 7ac9de5a8826fc95864ee4ba844eb8b5c9e71332
> # Parent  2c2e82469b8915c8153979cd89a970b7317f882d
> windows: add a method to convert Unix style command lines to Windows style

Queued, thanks.

> +        elif c == b'\\' and index + 1 < pathlen and path[index + 1] == b'$':
> +            # Skip '\', but only if it is escaping $
> +            res += b'$'
> +            index += 1

\-escape might be confusing since \ is the directory separator on Windows,
and a hook command is likely to contain backslashes.
Matt Harbison - June 29, 2018, 3:10 a.m.
On Thu, 28 Jun 2018 07:49:42 -0400, Yuya Nishihara <yuya@tcha.org> wrote:

> On Wed, 27 Jun 2018 08:44:26 -0400, Matt Harbison wrote:
>> # HG changeset patch
>> # User Matt Harbison <matt_harbison@yahoo.com>
>> # Date 1529817189 14400
>> #      Sun Jun 24 01:13:09 2018 -0400
>> # Node ID 7ac9de5a8826fc95864ee4ba844eb8b5c9e71332
>> # Parent  2c2e82469b8915c8153979cd89a970b7317f882d
>> windows: add a method to convert Unix style command lines to Windows  
>> style
>
> Queued, thanks.
>
>> +        elif c == b'\\' and index + 1 < pathlen and path[index + 1] ==  
>> b'$':
>> +            # Skip '\', but only if it is escaping $
>> +            res += b'$'
>> +            index += 1
>
> \-escape might be confusing since \ is the directory separator on  
> Windows, and a hook command is likely to contain backslashes.

Ugh.  I didn't even think about that, though I don't think path components  
starting with '$' are that common.

Thinking about this a bit more...

   - Maybe we should drop $$ -> $ too, since that's not what a Unix shell  
would do, and the point of this is portability.
   - Escaped '$' is a problem for hg -> cmd.exe -> sh.exe.  Consider `hook  
= echo \$var` in the config file.  This gets passed to cmd.exe as `echo  
$var`, but that doesn't change if something other than sh.exe runs.
   - I think translating `'` to `"` is the most useful thing to do, but  
that runs into the same problem.

What do you think?  Do we need an experimental knob to turn this off?  I'm  
assuming sh.exe is rare, and there aren't a whole lot of ways to hit this  
in practice.  I'd prefer it does the sensible thing out of the box.
Yuya Nishihara - June 29, 2018, 11:46 a.m.
On Thu, 28 Jun 2018 23:10:33 -0400, Matt Harbison wrote:
> On Thu, 28 Jun 2018 07:49:42 -0400, Yuya Nishihara <yuya@tcha.org> wrote:
> 
> > On Wed, 27 Jun 2018 08:44:26 -0400, Matt Harbison wrote:
> >> # HG changeset patch
> >> # User Matt Harbison <matt_harbison@yahoo.com>
> >> # Date 1529817189 14400
> >> #      Sun Jun 24 01:13:09 2018 -0400
> >> # Node ID 7ac9de5a8826fc95864ee4ba844eb8b5c9e71332
> >> # Parent  2c2e82469b8915c8153979cd89a970b7317f882d
> >> windows: add a method to convert Unix style command lines to Windows  
> >> style
> >
> > Queued, thanks.
> >
> >> +        elif c == b'\\' and index + 1 < pathlen and path[index + 1] ==  
> >> b'$':
> >> +            # Skip '\', but only if it is escaping $
> >> +            res += b'$'
> >> +            index += 1
> >
> > \-escape might be confusing since \ is the directory separator on  
> > Windows, and a hook command is likely to contain backslashes.
> 
> Ugh.  I didn't even think about that, though I don't think path components  
> starting with '$' are that common.
> 
> Thinking about this a bit more...
> 
>    - Maybe we should drop $$ -> $ too, since that's not what a Unix shell  
> would do, and the point of this is portability.

+1

>    - Escaped '$' is a problem for hg -> cmd.exe -> sh.exe.  Consider `hook  
> = echo \$var` in the config file.  This gets passed to cmd.exe as `echo  
> $var`, but that doesn't change if something other than sh.exe runs.
>    - I think translating `'` to `"` is the most useful thing to do, but  
> that runs into the same problem.
> 
> What do you think?  Do we need an experimental knob to turn this off?  I'm  
> assuming sh.exe is rare, and there aren't a whole lot of ways to hit this  
> in practice.  I'd prefer it does the sensible thing out of the box.

I think we should do as little shell emulation as possible because it can't
be perfect. Maybe we can leave \$foo unmodified (i.e. no $foo nor \%foo%) as
it is ambiguous?

I'm not so worrying about the hg->cmd.exe->sh thing, but if we add a config
knob, I think per-hook option is nicer.

Patch

diff --git a/mercurial/windows.py b/mercurial/windows.py
--- a/mercurial/windows.py
+++ b/mercurial/windows.py
@@ -12,6 +12,7 @@  import msvcrt
 import os
 import re
 import stat
+import string
 import sys
 
 from .i18n import _
@@ -253,6 +254,108 @@  normcasefallback = encoding.upperfallbac
 def samestat(s1, s2):
     return False
 
+def shelltocmdexe(path, env):
+    r"""Convert shell variables in the form $var and ${var} inside ``path``
+    to %var% form.  Existing Windows style variables are left unchanged.
+
+    The variables are limited to the given environment.  Unknown variables are
+    left unchanged.
+
+    >>> e = {b'var1': b'v1', b'var2': b'v2', b'var3': b'v3'}
+    >>> # Only valid values are expanded
+    >>> shelltocmdexe(b'cmd $var1 ${var2} %var3% $missing ${missing} %missing%',
+    ...               e)
+    'cmd %var1% %var2% %var3% $missing ${missing} %missing%'
+    >>> # Single quote prevents expansion, as does \$ escaping
+    >>> shelltocmdexe(b"cmd '$var1 ${var2} %var3%' \$var1 \${var2} \\", e)
+    "cmd '$var1 ${var2} %var3%' $var1 ${var2} \\"
+    >>> # $$ -> $, %% is not special, but can be the end and start of variables
+    >>> shelltocmdexe(b"cmd $$ %% %var1%%var2%", e)
+    'cmd $ %% %var1%%var2%'
+    >>> # No double substitution
+    >>> shelltocmdexe(b"$var1 %var1%", {b'var1': b'%var2%', b'var2': b'boom'})
+    '%var1% %var1%'
+    """
+    if b'$' not in path:
+        return path
+
+    varchars = pycompat.sysbytes(string.ascii_letters + string.digits) + b'_-'
+
+    res = b''
+    index = 0
+    pathlen = len(path)
+    while index < pathlen:
+        c = path[index]
+        if c == b'\'':   # no expansion within single quotes
+            path = path[index + 1:]
+            pathlen = len(path)
+            try:
+                index = path.index(b'\'')
+                res += b'\'' + path[:index + 1]
+            except ValueError:
+                res += c + path
+                index = pathlen - 1
+        elif c == b'%':  # variable
+            path = path[index + 1:]
+            pathlen = len(path)
+            try:
+                index = path.index(b'%')
+            except ValueError:
+                res += b'%' + path
+                index = pathlen - 1
+            else:
+                var = path[:index]
+                res += b'%' + var + b'%'
+        elif c == b'$':  # variable or '$$'
+            if path[index + 1:index + 2] == b'$':
+                res += c
+                index += 1
+            elif path[index + 1:index + 2] == b'{':
+                path = path[index + 2:]
+                pathlen = len(path)
+                try:
+                    index = path.index(b'}')
+                    var = path[:index]
+
+                    # See below for why empty variables are handled specially
+                    if env.get(var, '') != '':
+                        res += b'%' + var + b'%'
+                    else:
+                        res += b'${' + var + b'}'
+                except ValueError:
+                    res += b'${' + path
+                    index = pathlen - 1
+            else:
+                var = b''
+                index += 1
+                c = path[index:index + 1]
+                while c != b'' and c in varchars:
+                    var += c
+                    index += 1
+                    c = path[index:index + 1]
+                # Some variables (like HG_OLDNODE) may be defined, but have an
+                # empty value.  Those need to be skipped because when spawning
+                # cmd.exe to run the hook, it doesn't replace %VAR% for an empty
+                # VAR, and that really confuses things like revset expressions.
+                # OTOH, if it's left in Unix format and the hook runs sh.exe, it
+                # will substitute to an empty string, and everything is happy.
+                if env.get(var, '') != '':
+                    res += b'%' + var + b'%'
+                else:
+                    res += b'$' + var
+
+                if c != '':
+                    index -= 1
+        elif c == b'\\' and index + 1 < pathlen and path[index + 1] == b'$':
+            # Skip '\', but only if it is escaping $
+            res += b'$'
+            index += 1
+        else:
+            res += c
+
+        index += 1
+    return res
+
 # A sequence of backslashes is special iff it precedes a double quote:
 # - if there's an even number of backslashes, the double quote is not
 #   quoted (i.e. it ends the quoted region)