Patchwork [5,of,5,RFC,py3,support] run-tests: minimum changes required to make run-tests work in python3

login
register
mail settings
Submitter Augie Fackler
Date Sept. 19, 2013, 8:08 p.m.
Message ID <ff2da594a06def2e61f9.1379621294@arthedain.pit.corp.google.com>
Download mbox | patch
Permalink /patch/2546/
State Superseded
Headers show

Comments

Augie Fackler - Sept. 19, 2013, 8:08 p.m.
# HG changeset patch
# User Augie Fackler <raf@durin42.com>
# Date 1379621130 14400
#      Thu Sep 19 16:05:30 2013 -0400
# Node ID ff2da594a06def2e61f9e5e6226dcc27419a76fb
# Parent  cc547e41d77a130261625702e358307dde42dc39
run-tests: minimum changes required to make run-tests work in python3

This is just doing the dumbest thing possible and assumes test
input/output is all utf-8. This works for now, at least for the simple
tests I'm interested in trying to run, so I'm calling it progress
until I have to come back and figure out something smarter.

With these changes in place, all tests continue to pass on python2,
and you can run tests on python3 by running 2to3 over run-tests.py and
killdaemons.py. At that point, demandimport breaks on python3, causing
all tests to fail, but at least it's some progress.
Matt Mackall - Sept. 19, 2013, 8:36 p.m.
On Thu, 2013-09-19 at 16:08 -0400, Augie Fackler wrote:
> # HG changeset patch
> # User Augie Fackler <raf@durin42.com>
> # Date 1379621130 14400
> #      Thu Sep 19 16:05:30 2013 -0400
> # Node ID ff2da594a06def2e61f9e5e6226dcc27419a76fb
> # Parent  cc547e41d77a130261625702e358307dde42dc39
> run-tests: minimum changes required to make run-tests work in python3
> 
> This is just doing the dumbest thing possible and assumes test
> input/output is all utf-8. This works for now, at least for the simple
> tests I'm interested in trying to run, so I'm calling it progress
> until I have to come back and figure out something smarter.

Please tell me this doesn't mean subprocess can't return byte streams in
3.x.
Augie Fackler - Sept. 19, 2013, 8:37 p.m.
On Thu, Sep 19, 2013 at 4:36 PM, Matt Mackall <mpm@selenic.com> wrote:

> > This is just doing the dumbest thing possible and assumes test
> > input/output is all utf-8. This works for now, at least for the simple
> > tests I'm interested in trying to run, so I'm calling it progress
> > until I have to come back and figure out something smarter.
>
> Please tell me this doesn't mean subprocess can't return byte streams in
> 3.x.



I haven't looked at all. I'm literally doing the dumbest thing possible to
try and get to 'hg version' with a working testsuite.
Matt Mackall - Sept. 19, 2013, 8:53 p.m.
On Thu, 2013-09-19 at 16:37 -0400, Augie Fackler wrote:
> 
> On Thu, Sep 19, 2013 at 4:36 PM, Matt Mackall <mpm@selenic.com> wrote:
>         > This is just doing the dumbest thing possible and assumes
>         test
>         > input/output is all utf-8. This works for now, at least for
>         the simple
>         > tests I'm interested in trying to run, so I'm calling it
>         progress
>         > until I have to come back and figure out something smarter.
>         
>         
>         Please tell me this doesn't mean subprocess can't return byte
>         streams in
>         3.x.
> 
> 
> 
> I haven't looked at all. I'm literally doing the dumbest thing
> possible to try and get to 'hg version' with a working testsuite.

Ok. Let's learn a bit more before proceeding. A quick test here with
Py3.2 shows that subprocess pipe fds are in fact in byte mode.
Augie Fackler - Sept. 19, 2013, 10:06 p.m.
On Sep 19, 2013, at 4:53 PM, Matt Mackall <mpm@selenic.com> wrote:

> On Thu, 2013-09-19 at 16:37 -0400, Augie Fackler wrote:
>> 
>> On Thu, Sep 19, 2013 at 4:36 PM, Matt Mackall <mpm@selenic.com> wrote:
>>> This is just doing the dumbest thing possible and assumes
>>        test
>>> input/output is all utf-8. This works for now, at least for
>>        the simple
>>> tests I'm interested in trying to run, so I'm calling it
>>        progress
>>> until I have to come back and figure out something smarter.
>> 
>> 
>>        Please tell me this doesn't mean subprocess can't return byte
>>        streams in
>>        3.x.
>> 
>> 
>> 
>> I haven't looked at all. I'm literally doing the dumbest thing
>> possible to try and get to 'hg version' with a working testsuite.
> 
> Ok. Let's learn a bit more before proceeding. A quick test here with
> Py3.2 shows that subprocess pipe fds are in fact in byte mode.

Oh, dur. I forgot between writing the patch and responding what's going on here. What we're doing is manipulating the test content in string form, then encoding to bytes and writing, executing, then taking the bytes from the successful test run and converting it back to a string for feeding to difflib.

I think Python 3.4 will come with a difflib patch that adds support for bytes to difflib, which will be good progress, but for the mean time this at least lets us test hg using run-tests on 3.x and chip at getting something basic (probably 'hg version') working.

> 
> -- 
> Mathematics is the supreme nostalgia of our time.
> 
>

Patch

diff --git a/tests/run-tests.py b/tests/run-tests.py
--- a/tests/run-tests.py
+++ b/tests/run-tests.py
@@ -60,11 +60,24 @@ 
 
 processlock = threading.Lock()
 
-# subprocess._cleanup can race with any Popen.wait or Popen.poll on py24
-# http://bugs.python.org/issue1731717 for details. We shouldn't be producing
-# zombies but it's pretty harmless even if we do.
-if sys.version_info[1] < 5:
-    subprocess._cleanup = lambda: None
+if sys.version_info[0] >= 3:
+    def makebytes(d):
+        if isinstance(d, str):
+            return d.encode('utf-8')
+        return d
+    def makestr(d):
+        if isinstance(d, bytes):
+            return d.decode('utf-8')
+        return d
+else:
+    # subprocess._cleanup can race with any Popen.wait or Popen.poll on py24
+    # http://bugs.python.org/issue1731717 for details. We shouldn't be producing
+    # zombies but it's pretty harmless even if we do.
+    if sys.version_info[1] < 5:
+        subprocess._cleanup = lambda: None
+    def makebytes(d):
+        return d
+    makestr = makebytes
 
 closefds = os.name == 'posix'
 def Popen4(cmd, wd, timeout, env=None):
@@ -764,7 +777,7 @@ 
     fd, name = tempfile.mkstemp(suffix='hg-tst')
     try:
         for l in script:
-            os.write(fd, l)
+            os.write(fd, makebytes(l))
         os.close(fd)
 
         cmd = '%s "%s"' % (options.shell, name)
@@ -860,6 +873,7 @@ 
     if abort:
         raise KeyboardInterrupt()
 
+    output = makestr(output)
     for s, r in replacements:
         output = re.sub(s, r, output)
     return ret, output.splitlines(True)
@@ -994,7 +1008,7 @@ 
         # Save errors to a file for diagnosis
         f = open(err, "wb")
         for line in out:
-            f.write(line)
+            f.write(makebytes(line))
         f.close()
 
     if skipped: