Patchwork [RFC] setup.py: always build and install hg.exe on Windows

login
register
mail settings
Submitter Yuya Nishihara
Date Dec. 5, 2015, 11:54 a.m.
Message ID <20151205205400.4fb4e6d2a5c8e33530cbb908@tcha.org>
Download mbox | patch
Permalink /patch/11845/
State Not Applicable
Headers show

Comments

Yuya Nishihara - Dec. 5, 2015, 11:54 a.m.
On Fri, 04 Dec 2015 00:27:53 -0800, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc@gmail.com>
> # Date 1449217488 28800
> #      Fri Dec 04 00:24:48 2015 -0800
> # Node ID 6f7ae69f2a3a5686d56436d386974ec87769ddb2
> # Parent  71aa5a26162d6e4a165b68f07b331e3e2eedc117
> setup.py: always build and install hg.exe on Windows

Can't we use setuptools?

https://pythonhosted.org/setuptools/setuptools.html#eggsecutable-scripts


It seems hg.exe is created by "pip install".


> +class hgbuildscripts(build_scripts):
> +    def run(self):
> +        if os.name == 'nt':
> +            self.run_command('build_hgexe')
> +
> +        return build_scripts.run(self)
> +

> @@ -464,20 +479,28 @@ class hginstallscripts(install_scripts):
>              if b('\0') in data:
>                  continue
>  
>              data = data.replace(b('@LIBDIR@'), libdir.encode(libdir_escape))
>              fp = open(outfile, 'wb')
>              fp.write(data)
>              fp.close()
>  
> +        if os.name == 'nt':
> +            hgexecommand = self.get_finalized_command('build_hgexe')
> +            dest = os.path.join(self.install_dir, 'hg.exe')
> +            shutil.copyfile(hgexecommand.hgexepath, dest)
> +            mode = ((os.stat(dest)[stat.ST_MODE]) | 0555) & 07777
> +            os.chmod(dest, mode)

Just a nitpicky idea. As we have a custom build_scripts command, we can simply
copy hg.exe to build/scripts-2.7. After that, install_scripts will copy
everything under that directory.

+class hgbuildscripts(build_scripts):
+    def run(self):
+        build_scripts.run(self)
+        if os.name == 'nt':
+            self.run_command('build_hgexe')
+            self.mkpath(self.build_dir)
+            buildexe = self.get_finalized_command('build_hgexe')
+            self.copy_file(buildexe.hgexepath, self.build_dir)
Adrian Buehlmann - Dec. 5, 2015, 8:57 p.m.
On 2015-12-05 12:54, Yuya Nishihara wrote:
> On Fri, 04 Dec 2015 00:27:53 -0800, Gregory Szorc wrote:
>> # HG changeset patch
>> # User Gregory Szorc <gregory.szorc@gmail.com>
>> # Date 1449217488 28800
>> #      Fri Dec 04 00:24:48 2015 -0800
>> # Node ID 6f7ae69f2a3a5686d56436d386974ec87769ddb2
>> # Parent  71aa5a26162d6e4a165b68f07b331e3e2eedc117
>> setup.py: always build and install hg.exe on Windows
> 
> Can't we use setuptools?
> 
> https://pythonhosted.org/setuptools/setuptools.html#eggsecutable-scripts
> 

Perhaps.

But the exe which this (apparently?) creates looks like having nothing
to do with the one that Mercurial already can build from
mercurial/exewrapper.c, which embeds a few Mercurial-specific tricks
(e.g. [1]).

exewrapper.c produces a hg.exe which (I think) I specifically wanted to
be able to use for running Mercurial's testsuite on Windows. So it was
mostly a developer "tool".

I think we also tried to make it compatible with the HackableMercurial
package [2], which - unfortunately - wasn't that successful regarding
usage, as there - apparently - were some misunderstandings on how that
works.

I'm not sure what's exactly wanted here. The Windows Mercurial .msi
installer-package which Steve is building produces yet another hg.exe,
which is tied to a specific library.zip, created (I think) using py2exe.

Perhaps Steve may have some remarks. He used to be (and most likely
still is) an expert on that.

Gregory's patch looks harmless and useful to me, but I haven't tried it
myself.

[1] https://selenic.com/repo/hg/file/tip/mercurial/exewrapper.c#l131
[2] https://www.mercurial-scm.org/wiki/HackableMercurial

> diff --git a/setup.py b/setup.py
> --- a/setup.py
> +++ b/setup.py
> @@ -562,6 +562,12 @@ if py2exeloaded:
>      build.sub_commands.insert(0, ('build_hgextindex', None))
>      # put dlls in sub directory so that they won't pollute PATH
>      extra['zipfile'] = 'lib/library.zip'
> +if 'FORCE_SETUPTOOLS' in os.environ:
> +    extra['entry_points'] = {
> +        'console_scripts': [
> +            'hg = mercurial.dispatch:run',  # BUG skips demandimport, setbinary
> +        ],
> +    }
>  
>  if os.name == 'nt':
>      # Windows binary file versions for exe/dll files must have the
> 
> It seems hg.exe is created by "pip install".
> 
>
Gregory Szorc - Dec. 6, 2015, 12:17 a.m.
On Sat, Dec 5, 2015 at 12:57 PM, Adrian Buehlmann <adrian@cadifra.com>
wrote:

> On 2015-12-05 12:54, Yuya Nishihara wrote:
> > On Fri, 04 Dec 2015 00:27:53 -0800, Gregory Szorc wrote:
> >> # HG changeset patch
> >> # User Gregory Szorc <gregory.szorc@gmail.com>
> >> # Date 1449217488 28800
> >> #      Fri Dec 04 00:24:48 2015 -0800
> >> # Node ID 6f7ae69f2a3a5686d56436d386974ec87769ddb2
> >> # Parent  71aa5a26162d6e4a165b68f07b331e3e2eedc117
> >> setup.py: always build and install hg.exe on Windows
> >
> > Can't we use setuptools?
> >
> > https://pythonhosted.org/setuptools/setuptools.html#eggsecutable-scripts
> >
>

While setuptools supports "eggsecutables," I'm pretty sure that modern
Python packaging conventions frown on anything related to eggs and I'm
guessing we shouldn't even consider using eggsecutables.

FWIW, I attempted to use setuptools earlier this year and ran into issues.
https://selenic.com/pipermail/mercurial-devel/2015-February/066236.html


> Perhaps.
>
> But the exe which this (apparently?) creates looks like having nothing
> to do with the one that Mercurial already can build from
> mercurial/exewrapper.c, which embeds a few Mercurial-specific tricks
> (e.g. [1]).
>
> exewrapper.c produces a hg.exe which (I think) I specifically wanted to
> be able to use for running Mercurial's testsuite on Windows. So it was
> mostly a developer "tool".
>
>
Another benefit of hg.exe is it isn't a batch file. mpm made references to
issues with `hg` as a batch script at
https://selenic.com/pipermail/mercurial-devel/2015-October/074430.html.


> I think we also tried to make it compatible with the HackableMercurial
> package [2], which - unfortunately - wasn't that successful regarding
> usage, as there - apparently - were some misunderstandings on how that
> works.
>
> I'm not sure what's exactly wanted here. The Windows Mercurial .msi
> installer-package which Steve is building produces yet another hg.exe,
> which is tied to a specific library.zip, created (I think) using py2exe.
>

Correct. This hg.exe from the MSI loads the self-contained Mercurial+Python
distribution created by py2exe.

The goal of this patch is to enable Windows users to `pip install
Mercurial` and get a proper Mercurial package. It will also enable us to
produce wheels for Windows, which we can upload to PyPI so Windows users
don't need a compiler to install Mercurial via pip. Mozilla cares about
wheel distribution because it is much easier for us to manage Python
packages on our Windows automation via pip than with MSI installers.


>
> Perhaps Steve may have some remarks. He used to be (and most likely
> still is) an expert on that.
>
> Gregory's patch looks harmless and useful to me, but I haven't tried it
> myself.
>
> [1] https://selenic.com/repo/hg/file/tip/mercurial/exewrapper.c#l131
> [2] https://www.mercurial-scm.org/wiki/HackableMercurial




>
>
> > diff --git a/setup.py b/setup.py
> > --- a/setup.py
> > +++ b/setup.py
> > @@ -562,6 +562,12 @@ if py2exeloaded:
> >      build.sub_commands.insert(0, ('build_hgextindex', None))
> >      # put dlls in sub directory so that they won't pollute PATH
> >      extra['zipfile'] = 'lib/library.zip'
> > +if 'FORCE_SETUPTOOLS' in os.environ:
> > +    extra['entry_points'] = {
> > +        'console_scripts': [
> > +            'hg = mercurial.dispatch:run',  # BUG skips demandimport,
> setbinary
> > +        ],
> > +    }
> >
> >  if os.name == 'nt':
> >      # Windows binary file versions for exe/dll files must have the
> >
> > It seems hg.exe is created by "pip install".
> >
> >
>
Pierre-Yves David - Dec. 6, 2015, 6:05 a.m.
On 12/05/2015 04:17 PM, Gregory Szorc wrote:
> On Sat, Dec 5, 2015 at 12:57 PM, Adrian Buehlmann <adrian@cadifra.com
> <mailto:adrian@cadifra.com>> wrote:
>
>     On 2015-12-05 12:54, Yuya Nishihara wrote:
>     > On Fri, 04 Dec 2015 00:27:53 -0800, Gregory Szorc wrote:
>     >> # HG changeset patch
>     >> # User Gregory Szorc <gregory.szorc@gmail.com <mailto:gregory.szorc@gmail.com>>
>     >> # Date 1449217488 28800
>     >> #      Fri Dec 04 00:24:48 2015 -0800
>     >> # Node ID 6f7ae69f2a3a5686d56436d386974ec87769ddb2
>     >> # Parent  71aa5a26162d6e4a165b68f07b331e3e2eedc117
>     >> setup.py: always build and install hg.exe on Windows
>     >
>     > Can't we use setuptools?
>     >
>     >https://pythonhosted.org/setuptools/setuptools.html#eggsecutable-scripts
>     >
>
>
> While setuptools supports "eggsecutables," I'm pretty sure that modern
> Python packaging conventions frown on anything related to eggs and I'm
> guessing we shouldn't even consider using eggsecutables.

Please do not use eggs, ever.

> FWIW, I attempted to use setuptools earlier this year and ran into
> issues.
> https://selenic.com/pipermail/mercurial-devel/2015-February/066236.html
>
>     Perhaps.
>
>     But the exe which this (apparently?) creates looks like having nothing
>     to do with the one that Mercurial already can build from
>     mercurial/exewrapper.c, which embeds a few Mercurial-specific tricks
>     (e.g. [1]).
>
>     exewrapper.c produces a hg.exe which (I think) I specifically wanted to
>     be able to use for running Mercurial's testsuite on Windows. So it was
>     mostly a developer "tool".
>
>
> Another benefit of hg.exe is it isn't a batch file. mpm made references
> to issues with `hg` as a batch script at
> https://selenic.com/pipermail/mercurial-devel/2015-October/074430.html.
>
>     I think we also tried to make it compatible with the HackableMercurial
>     package [2], which - unfortunately - wasn't that successful regarding
>     usage, as there - apparently - were some misunderstandings on how that
>     works.
>
>     I'm not sure what's exactly wanted here. The Windows Mercurial .msi
>     installer-package which Steve is building produces yet another hg.exe,
>     which is tied to a specific library.zip, created (I think) using py2exe.
>
>
> Correct. This hg.exe from the MSI loads the self-contained
> Mercurial+Python distribution created by py2exe.
>
> The goal of this patch is to enable Windows users to `pip install
> Mercurial` and get a proper Mercurial package. It will also enable us to
> produce wheels for Windows, which we can upload to PyPI so Windows users
> don't need a compiler to install Mercurial via pip. Mozilla cares about
> wheel distribution because it is much easier for us to manage Python
> packages on our Windows automation via pip than with MSI installers.
>
>
>     Perhaps Steve may have some remarks. He used to be (and most likely
>     still is) an expert on that.
>
>     Gregory's patch looks harmless and useful to me, but I haven't tried it
>     myself.

Adrian, can you try it? I'll queue it if you report success.
Pierre-Yves David - Dec. 6, 2015, 8:22 a.m.
On 12/05/2015 10:05 PM, Pierre-Yves David wrote:
>
>
> On 12/05/2015 04:17 PM, Gregory Szorc wrote:
>> On Sat, Dec 5, 2015 at 12:57 PM, Adrian Buehlmann <adrian@cadifra.com
>> <mailto:adrian@cadifra.com>> wrote:
>>> Perhaps Steve may have some remarks. He used to be (and most likely
>>> still is) an expert on that.
>>>
>>> Gregory's patch looks harmless and useful to me, but I haven't tried it
>>>
>>> myself.
>
> Adrian, can you try it? I'll queue it if you report success.

nervermind, there is a V2 in flight and yuya is taking care of it.
Adrian Buehlmann - Dec. 6, 2015, 8:50 a.m.
On 2015-12-06 07:05, Pierre-Yves David wrote:
> Adrian, can you try it? I'll queue it if you report success.

Hmm. My Mercurial package-/installer-building environment - which might
be useful for revealing potential unknown bad side effects - is
suspected to be horribly out of sync currently & has been unused for
quite a while now. My (rather limited) understanding of it faded away
accordingly, and would have to be refreshed - which I have no plans to
do ATM, sorry. But I'm happy to see others still contributing and using
Mercurial.

Since Steve appears to still be building Windows packages for Mercurial
and TortoiseHg, he might be able to provide first-hand feedback (if you
prefer having such kind of feedback before queueing...). Perhaps, just
wait a bit more?

Other than that, I think Gregory can surely be trusted that the patch
does what it claims...

Patch

diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -562,6 +562,12 @@  if py2exeloaded:
     build.sub_commands.insert(0, ('build_hgextindex', None))
     # put dlls in sub directory so that they won't pollute PATH
     extra['zipfile'] = 'lib/library.zip'
+if 'FORCE_SETUPTOOLS' in os.environ:
+    extra['entry_points'] = {
+        'console_scripts': [
+            'hg = mercurial.dispatch:run',  # BUG skips demandimport, setbinary
+        ],
+    }
 
 if os.name == 'nt':
     # Windows binary file versions for exe/dll files must have the