Patchwork py3: write out hgextindex as bytes in setup.py

login
register
mail settings
Submitter Matt Harbison
Date April 5, 2019, 10:38 p.m.
Message ID <d8aef987cc88a5abead0.1554503934@Envy>
Download mbox | patch
Permalink /patch/39522/
State Accepted
Headers show

Comments

Matt Harbison - April 5, 2019, 10:38 p.m.
# HG changeset patch
# User Matt Harbison <matt_harbison@yahoo.com>
# Date 1554503803 14400
#      Fri Apr 05 18:36:43 2019 -0400
# Node ID d8aef987cc88a5abead0c9cecf5e14066f161968
# Parent  a975821d0938615b8cb235f12cc6461a42dcfbd8
py3: write out hgextindex as bytes in setup.py

I hit this trying to build the py2exe target using python3, just to see what
would happen.  After commenting out `py2exe.Distribution` in setup.py and
pointing to a local copy of py2exe that supports python3[1], it complained that
`out` was bytes, not str.

[1] https://github.com/albertosottile/py2exe/releases/tag/v0.9.3.0
Matt Harbison - April 5, 2019, 10:44 p.m.
On Fri, 05 Apr 2019 18:38:54 -0400, Matt Harbison <mharbison72@gmail.com>  
wrote:

> # HG changeset patch
> # User Matt Harbison <matt_harbison@yahoo.com>
> # Date 1554503803 14400
> #      Fri Apr 05 18:36:43 2019 -0400
> # Node ID d8aef987cc88a5abead0c9cecf5e14066f161968
> # Parent  a975821d0938615b8cb235f12cc6461a42dcfbd8
> py3: write out hgextindex as bytes in setup.py

There are subsequent problems that need to be resolved with py2exe.   
First, it's
complaining about py2 syntax in the vendored `concurrent` package.   
Obviously we
don't want that picked up in py3, and there's logic in setup.py to only  
add it
under py2.  But if added explicitly to the exclude list, the build fails
complaining there's no such package.

Commenting that py2 syntax out, it then dies inside the custom module  
loader at
`spec = finder.find_spec(..)`, complaining:

     'VendorImporter' object has no attribute 'find_spec'

Wrapping that in an exception handler that does nothing allows the build to
complete, but hg.exe can't load dispatch.py from the zip file.  There's
a TODO in the custom loader to support loading from zip files, but it seems
to bail out before that point because `finder.find_spec()` returns None.   
The
loader can't be removed, because there's still something very early that  
needs
to be treated as bytes.

Presumably there's already a loader that knows how to read zip files that  
we
could just delegate to somehow, but I really don't have a good  
understanding of
how all these pieces fit together.  `sys.meta_path` at this point is:

     [mercurial.hgpathentryfinder, _frozen_importlib.BuiltinImporter,
      _frozen_importlib.FrozenImporter,  
_frozen_importlib_external.PathFinder]
Gregory Szorc - April 5, 2019, 10:59 p.m.
> On Apr 5, 2019, at 15:44, Matt Harbison <mharbison72@gmail.com> wrote:
> 
>> On Fri, 05 Apr 2019 18:38:54 -0400, Matt Harbison <mharbison72@gmail.com> wrote:
>> 
>> # HG changeset patch
>> # User Matt Harbison <matt_harbison@yahoo.com>
>> # Date 1554503803 14400
>> #      Fri Apr 05 18:36:43 2019 -0400
>> # Node ID d8aef987cc88a5abead0c9cecf5e14066f161968
>> # Parent  a975821d0938615b8cb235f12cc6461a42dcfbd8
>> py3: write out hgextindex as bytes in setup.py
> 
> There are subsequent problems that need to be resolved with py2exe.  First, it's
> complaining about py2 syntax in the vendored `concurrent` package.  Obviously we
> don't want that picked up in py3, and there's logic in setup.py to only add it
> under py2.  But if added explicitly to the exclude list, the build fails
> complaining there's no such package.
> 
> Commenting that py2 syntax out, it then dies inside the custom module loader at
> `spec = finder.find_spec(..)`, complaining:
> 
>    'VendorImporter' object has no attribute 'find_spec'
> 
> Wrapping that in an exception handler that does nothing allows the build to
> complete, but hg.exe can't load dispatch.py from the zip file.  There's
> a TODO in the custom loader to support loading from zip files, but it seems
> to bail out before that point because `finder.find_spec()` returns None.  The
> loader can't be removed, because there's still something very early that needs
> to be treated as bytes.
> 
> Presumably there's already a loader that knows how to read zip files that we
> could just delegate to somehow, but I really don't have a good understanding of
> how all these pieces fit together.  `sys.meta_path` at this point is:
> 
>    [mercurial.hgpathentryfinder, _frozen_importlib.BuiltinImporter,
>     _frozen_importlib.FrozenImporter, _frozen_importlib_external.PathFinder]

The source transforming module importer is going to be “fun” to get working for py2exe. That’s because I believe py2exe is performing its own pyc compilation so the zip has only pyc files. I have doubts that the module compilation includes our custom source transformer. But I could be wrong.

We have a few paths forward...

a) coerce py2exe to work with our custom module importer. This entails either compiling transformed source into bytecode and teaching our importer to load it from the zip file. Or putting .py files in the zip and making our module importer work with that.

b) get rid of the source transforming module importer. (I was kinda hoping we’d be going down this path already. But doing so entails b’’ everywhere and we were ratholing in mass rewriting the repo.)

c) give up on py2exe and use PyOxidizer. (I’m still hacking on this, albeit slowly.) PyOxidizer is superior because it is faster and will be a self-contained binary. Mercurial is very much my main test case for PyOxidizer and I’ll add features to PyOxidizer to make it work with Mercurial.
Yuya Nishihara - April 6, 2019, 1:02 a.m.
On Fri, 05 Apr 2019 18:38:54 -0400, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison <matt_harbison@yahoo.com>
> # Date 1554503803 14400
> #      Fri Apr 05 18:36:43 2019 -0400
> # Node ID d8aef987cc88a5abead0c9cecf5e14066f161968
> # Parent  a975821d0938615b8cb235f12cc6461a42dcfbd8
> py3: write out hgextindex as bytes in setup.py

Queued, thanks.
Matt Harbison - April 6, 2019, 3:25 a.m.
On Fri, 05 Apr 2019 18:59:42 -0400, Gregory Szorc  
<gregory.szorc@gmail.com> wrote:

>
> The source transforming module importer is going to be “fun” to get  
> working for py2exe. That’s because I believe py2exe is performing its  
> own pyc compilation so the zip has only pyc files. I have doubts that  
> the module compilation includes our custom source transformer. But I  
> could be wrong.
>
> We have a few paths forward...
>
> a) coerce py2exe to work with our custom module importer. This entails  
> either compiling transformed source into bytecode and teaching our  
> importer to load it from the zip file. Or putting .py files in the zip  
> and making our module importer work with that.
>
> b) get rid of the source transforming module importer. (I was kinda  
> hoping we’d be going down this path already. But doing so entails b’’  
> everywhere and we were ratholing in mass rewriting the repo.)
>
> c) give up on py2exe and use PyOxidizer. (I’m still hacking on this,  
> albeit slowly.) PyOxidizer is superior because it is faster and will be  
> a self-contained binary. Mercurial is very much my main test case for  
> PyOxidizer and I’ll add features to PyOxidizer to make it work with  
> Mercurial.

OK, thanks.  I didn't know anything about PyOxidizer- it sounds promising!

I'll probably leave py2exe alone then, as that's well beyond my level of  
knowledge and it sounds like it's more hassle than it's worth.  I was kind  
of interested in how easily it would integrate into the TortoiseHG build-  
I don't have the custom PyQt build that works with py2, so I thought maybe  
working through the py3 issues would provide a platform to test packaging  
changes.

Additionally, it errors out on my laptop when installing the wheel:

C:\Program Files (x86)\Microsoft Visual  
Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\binHostX64\x64\cl.exe  
/c /nologo /Ox /MD /W3 /GS- /DNDEBUG -DPYTHONDLL=\"PYTHON27.DLL\"  
-DPYTHONCOM=\"pythoncom27.dll\" -IC:\Python27\include  
-IC:\Users\Matt\hg\build\venv-inno-x64\PC /Tcsource/start.c  
/Fobuild\temp.win-amd64-2.7\Release\source/start.obj start.c
C:\Program Files (x86)\Windows  
Kits\10\include\10.0.17763.0\ucrt\stdio.h(1933): warning C4005:  
'snprintf': macro redefinition
c:\users\matt\hg\build\py2exe-0.6.9\source\Python-dynload.h(30): note: see  
previous definition of 'snprintf'
C:\Program Files (x86)\Windows  
Kits\10\include\10.0.17763.0\ucrt\stdio.h(1935): fatal error C1189:  
#error:  Macro definition of snprintf conflicts with Standard Library  
function declaration
error: command 'C:\\Program Files (x86)\\Microsoft Visual  
Studio\\2017\\Professional\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX64\\x64\\cl.exe'  
failed with exit status 2

It built successfully at work earlier today, so I'm not sure what the  
difference is here- possibly the SDK version?

Patch

diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -583,9 +583,9 @@  class buildhgextindex(Command):
         if err or returncode != 0:
             raise DistutilsExecError(err)
 
-        with open(self._indexfilename, 'w') as f:
-            f.write('# this file is autogenerated by setup.py\n')
-            f.write('docs = ')
+        with open(self._indexfilename, 'wb') as f:
+            f.write(b'# this file is autogenerated by setup.py\n')
+            f.write(b'docs = ')
             f.write(out)
 
 class buildhgexe(build_ext):