| Submitter | Matt Harbison |
|---|---|
| Date | April 5, 2019, 10:38 p.m. |
| Message ID | <d8aef987cc88a5abead0.1554503934@Envy> |
| Download | mbox | patch |
| Permalink | /patch/39522/ |
| State | Accepted |
| Headers | show |
Comments
On Fri, 05 Apr 2019 18:38:54 -0400, Matt Harbison <mharbison72@gmail.com> wrote: > # HG changeset patch > # User Matt Harbison <matt_harbison@yahoo.com> > # Date 1554503803 14400 > # Fri Apr 05 18:36:43 2019 -0400 > # Node ID d8aef987cc88a5abead0c9cecf5e14066f161968 > # Parent a975821d0938615b8cb235f12cc6461a42dcfbd8 > py3: write out hgextindex as bytes in setup.py There are subsequent problems that need to be resolved with py2exe. First, it's complaining about py2 syntax in the vendored `concurrent` package. Obviously we don't want that picked up in py3, and there's logic in setup.py to only add it under py2. But if added explicitly to the exclude list, the build fails complaining there's no such package. Commenting that py2 syntax out, it then dies inside the custom module loader at `spec = finder.find_spec(..)`, complaining: 'VendorImporter' object has no attribute 'find_spec' Wrapping that in an exception handler that does nothing allows the build to complete, but hg.exe can't load dispatch.py from the zip file. There's a TODO in the custom loader to support loading from zip files, but it seems to bail out before that point because `finder.find_spec()` returns None. The loader can't be removed, because there's still something very early that needs to be treated as bytes. Presumably there's already a loader that knows how to read zip files that we could just delegate to somehow, but I really don't have a good understanding of how all these pieces fit together. `sys.meta_path` at this point is: [mercurial.hgpathentryfinder, _frozen_importlib.BuiltinImporter, _frozen_importlib.FrozenImporter, _frozen_importlib_external.PathFinder]
> On Apr 5, 2019, at 15:44, Matt Harbison <mharbison72@gmail.com> wrote: > >> On Fri, 05 Apr 2019 18:38:54 -0400, Matt Harbison <mharbison72@gmail.com> wrote: >> >> # HG changeset patch >> # User Matt Harbison <matt_harbison@yahoo.com> >> # Date 1554503803 14400 >> # Fri Apr 05 18:36:43 2019 -0400 >> # Node ID d8aef987cc88a5abead0c9cecf5e14066f161968 >> # Parent a975821d0938615b8cb235f12cc6461a42dcfbd8 >> py3: write out hgextindex as bytes in setup.py > > There are subsequent problems that need to be resolved with py2exe. First, it's > complaining about py2 syntax in the vendored `concurrent` package. Obviously we > don't want that picked up in py3, and there's logic in setup.py to only add it > under py2. But if added explicitly to the exclude list, the build fails > complaining there's no such package. > > Commenting that py2 syntax out, it then dies inside the custom module loader at > `spec = finder.find_spec(..)`, complaining: > > 'VendorImporter' object has no attribute 'find_spec' > > Wrapping that in an exception handler that does nothing allows the build to > complete, but hg.exe can't load dispatch.py from the zip file. There's > a TODO in the custom loader to support loading from zip files, but it seems > to bail out before that point because `finder.find_spec()` returns None. The > loader can't be removed, because there's still something very early that needs > to be treated as bytes. > > Presumably there's already a loader that knows how to read zip files that we > could just delegate to somehow, but I really don't have a good understanding of > how all these pieces fit together. `sys.meta_path` at this point is: > > [mercurial.hgpathentryfinder, _frozen_importlib.BuiltinImporter, > _frozen_importlib.FrozenImporter, _frozen_importlib_external.PathFinder] The source transforming module importer is going to be “fun” to get working for py2exe. That’s because I believe py2exe is performing its own pyc compilation so the zip has only pyc files. I have doubts that the module compilation includes our custom source transformer. But I could be wrong. We have a few paths forward... a) coerce py2exe to work with our custom module importer. This entails either compiling transformed source into bytecode and teaching our importer to load it from the zip file. Or putting .py files in the zip and making our module importer work with that. b) get rid of the source transforming module importer. (I was kinda hoping we’d be going down this path already. But doing so entails b’’ everywhere and we were ratholing in mass rewriting the repo.) c) give up on py2exe and use PyOxidizer. (I’m still hacking on this, albeit slowly.) PyOxidizer is superior because it is faster and will be a self-contained binary. Mercurial is very much my main test case for PyOxidizer and I’ll add features to PyOxidizer to make it work with Mercurial.
On Fri, 05 Apr 2019 18:38:54 -0400, Matt Harbison wrote: > # HG changeset patch > # User Matt Harbison <matt_harbison@yahoo.com> > # Date 1554503803 14400 > # Fri Apr 05 18:36:43 2019 -0400 > # Node ID d8aef987cc88a5abead0c9cecf5e14066f161968 > # Parent a975821d0938615b8cb235f12cc6461a42dcfbd8 > py3: write out hgextindex as bytes in setup.py Queued, thanks.
On Fri, 05 Apr 2019 18:59:42 -0400, Gregory Szorc <gregory.szorc@gmail.com> wrote: > > The source transforming module importer is going to be “fun” to get > working for py2exe. That’s because I believe py2exe is performing its > own pyc compilation so the zip has only pyc files. I have doubts that > the module compilation includes our custom source transformer. But I > could be wrong. > > We have a few paths forward... > > a) coerce py2exe to work with our custom module importer. This entails > either compiling transformed source into bytecode and teaching our > importer to load it from the zip file. Or putting .py files in the zip > and making our module importer work with that. > > b) get rid of the source transforming module importer. (I was kinda > hoping we’d be going down this path already. But doing so entails b’’ > everywhere and we were ratholing in mass rewriting the repo.) > > c) give up on py2exe and use PyOxidizer. (I’m still hacking on this, > albeit slowly.) PyOxidizer is superior because it is faster and will be > a self-contained binary. Mercurial is very much my main test case for > PyOxidizer and I’ll add features to PyOxidizer to make it work with > Mercurial. OK, thanks. I didn't know anything about PyOxidizer- it sounds promising! I'll probably leave py2exe alone then, as that's well beyond my level of knowledge and it sounds like it's more hassle than it's worth. I was kind of interested in how easily it would integrate into the TortoiseHG build- I don't have the custom PyQt build that works with py2, so I thought maybe working through the py3 issues would provide a platform to test packaging changes. Additionally, it errors out on my laptop when installing the wheel: C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\VC\Tools\MSVC\14.16.27023\binHostX64\x64\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -DPYTHONDLL=\"PYTHON27.DLL\" -DPYTHONCOM=\"pythoncom27.dll\" -IC:\Python27\include -IC:\Users\Matt\hg\build\venv-inno-x64\PC /Tcsource/start.c /Fobuild\temp.win-amd64-2.7\Release\source/start.obj start.c C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\stdio.h(1933): warning C4005: 'snprintf': macro redefinition c:\users\matt\hg\build\py2exe-0.6.9\source\Python-dynload.h(30): note: see previous definition of 'snprintf' C:\Program Files (x86)\Windows Kits\10\include\10.0.17763.0\ucrt\stdio.h(1935): fatal error C1189: #error: Macro definition of snprintf conflicts with Standard Library function declaration error: command 'C:\\Program Files (x86)\\Microsoft Visual Studio\\2017\\Professional\\VC\\Tools\\MSVC\\14.16.27023\\bin\\HostX64\\x64\\cl.exe' failed with exit status 2 It built successfully at work earlier today, so I'm not sure what the difference is here- possibly the SDK version?
Patch
diff --git a/setup.py b/setup.py --- a/setup.py +++ b/setup.py @@ -583,9 +583,9 @@ class buildhgextindex(Command): if err or returncode != 0: raise DistutilsExecError(err) - with open(self._indexfilename, 'w') as f: - f.write('# this file is autogenerated by setup.py\n') - f.write('docs = ') + with open(self._indexfilename, 'wb') as f: + f.write(b'# this file is autogenerated by setup.py\n') + f.write(b'docs = ') f.write(out) class buildhgexe(build_ext):