Patchwork [2,of,2,V2] setup: use libpythonX.Y as library name when building with MinGW

login
register
mail settings
Submitter Gregory Szorc
Date April 27, 2016, 4:46 p.m.
Message ID <9d064dc961587258355c.1461775619@ubuntu-vm-main>
Download mbox | patch
Permalink /patch/14807/
State Superseded
Headers show

Comments

Gregory Szorc - April 27, 2016, 4:46 p.m.
# HG changeset patch
# User Gregory Szorc <gregory.szorc@gmail.com>
# Date 1461775344 25200
#      Wed Apr 27 09:42:24 2016 -0700
# Branch stable
# Node ID 9d064dc961587258355c53333002a763da460f2a
# Parent  5b7f85ccc5851871997f0383331f2982c6db1f43
setup: use libpythonX.Y as library name when building with MinGW

Attempting to build Mercurial from source using MinGW from
msys2 on Windows produces a hg.exe that attempts to load e.g.
python27.dll. MinGW prefixes its library name with "lib" and
adds a period between the major and minor versions. e.g.
"libpython2.7.dll."

Before this patch, hg.exe files in a MinGW environment would
either fail to find a Python DLL or would attempt to load a
non-MinGW DLL, which would summarily explode. Either way,
hg.exe wouldn't work.

This patch changes builds performed with a MinGW Python to detect
"libpythonX.Y" and use it in hg.exe.

If building with non-MinGW Python, we should continue to use
"pythonXY" as the library name, as there should not be a
"libpythonX.Y.dll" file next to sys.executable. This will enable
people to use MinGW as a dev/shell environment while continuing
to use a non-MinGW Python.
Adrian Buehlmann - April 27, 2016, 6:14 p.m.
On 2016-04-27 18:46, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc@gmail.com>
> # Date 1461775344 25200
> #      Wed Apr 27 09:42:24 2016 -0700
> # Branch stable
> # Node ID 9d064dc961587258355c53333002a763da460f2a
> # Parent  5b7f85ccc5851871997f0383331f2982c6db1f43
> setup: use libpythonX.Y as library name when building with MinGW
> 
> Attempting to build Mercurial from source using MinGW from
> msys2 on Windows produces a hg.exe that attempts to load e.g.
> python27.dll. MinGW prefixes its library name with "lib" and
> adds a period between the major and minor versions. e.g.
> "libpython2.7.dll."
> 
> Before this patch, hg.exe files in a MinGW environment would
> either fail to find a Python DLL or would attempt to load a
> non-MinGW DLL, which would summarily explode. Either way,
> hg.exe wouldn't work.
> 
> This patch changes builds performed with a MinGW Python to detect
> "libpythonX.Y" and use it in hg.exe.
> 
> If building with non-MinGW Python, we should continue to use
> "pythonXY" as the library name, as there should not be a
> "libpythonX.Y.dll" file next to sys.executable. This will enable
> people to use MinGW as a dev/shell environment while continuing
> to use a non-MinGW Python.
> 
> diff --git a/setup.py b/setup.py
> --- a/setup.py
> +++ b/setup.py
> @@ -364,18 +364,38 @@ class buildhgexe(build_ext):
>      description = 'compile hg.exe from mercurial/exewrapper.c'
>  
>      def build_extensions(self):
>          if os.name != 'nt':
>              return
>          if isinstance(self.compiler, HackedMingw32CCompiler):
>              self.compiler.compiler_so = self.compiler.compiler # no -mdll
>              self.compiler.dll_libraries = [] # no -lmsrvc90
> +
>          hv = sys.hexversion
> -        pythonlib = 'python%d%d' % (hv >> 24, (hv >> 16) & 0xff)
> +        maj = '%s' % (hv >> 24,)
> +        min = '%s' % ((hv >> 16) & 0xff,)
> +
> +        # Most installs (including the official CPython distribution) have a
> +        # pythonXY.dll somewhere (probably on the PATH).
> +        libpattern = 'python%s%s'
> +
> +        # MinGW puts a libpythonX.Y.dll in the same directory as python.exe.
> +        # If it exists, we use it.
> +        libpythonpath = os.path.join(os.path.dirname(sys.executable),
> +                                     'libpython%s.%s.dll' % (maj, min))
> +        if os.path.exists(libpythonpath):
> +            libpattern = 'libpython%s.%s'
> +
> +        # We don't search for libpythonX.Y.dll on PATH because it is valid to
> +        # run non-MinGW Python from within MinGW and we don't want false
> +        # positives.
> +
> +        pythonlib = libpattern % (maj, min)
> +        log.info('using %s as Python library name' % pythonlib)
>          with open('mercurial/hgpythonlib.h', 'wb') as f:
>              f.write('/* this file is autogenerated by setup.py */\n')
>              f.write('#define HGPYTHONLIB "%s"\n' % pythonlib)
>          objects = self.compiler.compile(['mercurial/exewrapper.c'],
>                                           output_dir=self.build_temp)
>          dir = os.path.dirname(self.get_ext_fullpath('dummy'))
>          target = os.path.join(dir, 'hg')
>          self.compiler.link_executable(objects, target,

Perhaps, we can be even more clever. See for example:

$ python
Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.dllhandle
503316480L
>>> import ctypes
>>> _kernel32 = ctypes.windll.kernel32
>>> size = 600
>>> buf = ctypes.create_string_buffer(size + 1)
>>> len = _kernel32.GetModuleFileNameA(sys.dllhandle, ctypes.byref(buf), size)
>>> len
32
>>> buf.value
'C:\\Windows\\system32\\python27.dll'
>>>

Would it be possible to extract the name of the python dll this way?

(see also executablepath in mercurial/win32.py for an example call of
GetModuleFileNameA)
Gregory Szorc - April 27, 2016, 8:01 p.m.
On Wed, Apr 27, 2016 at 11:14 AM, Adrian Buehlmann <adrian@cadifra.com>
wrote:

> On 2016-04-27 18:46, Gregory Szorc wrote:
> > # HG changeset patch
> > # User Gregory Szorc <gregory.szorc@gmail.com>
> > # Date 1461775344 25200
> > #      Wed Apr 27 09:42:24 2016 -0700
> > # Branch stable
> > # Node ID 9d064dc961587258355c53333002a763da460f2a
> > # Parent  5b7f85ccc5851871997f0383331f2982c6db1f43
> > setup: use libpythonX.Y as library name when building with MinGW
> >
> > Attempting to build Mercurial from source using MinGW from
> > msys2 on Windows produces a hg.exe that attempts to load e.g.
> > python27.dll. MinGW prefixes its library name with "lib" and
> > adds a period between the major and minor versions. e.g.
> > "libpython2.7.dll."
> >
> > Before this patch, hg.exe files in a MinGW environment would
> > either fail to find a Python DLL or would attempt to load a
> > non-MinGW DLL, which would summarily explode. Either way,
> > hg.exe wouldn't work.
> >
> > This patch changes builds performed with a MinGW Python to detect
> > "libpythonX.Y" and use it in hg.exe.
> >
> > If building with non-MinGW Python, we should continue to use
> > "pythonXY" as the library name, as there should not be a
> > "libpythonX.Y.dll" file next to sys.executable. This will enable
> > people to use MinGW as a dev/shell environment while continuing
> > to use a non-MinGW Python.
> >
> > diff --git a/setup.py b/setup.py
> > --- a/setup.py
> > +++ b/setup.py
> > @@ -364,18 +364,38 @@ class buildhgexe(build_ext):
> >      description = 'compile hg.exe from mercurial/exewrapper.c'
> >
> >      def build_extensions(self):
> >          if os.name != 'nt':
> >              return
> >          if isinstance(self.compiler, HackedMingw32CCompiler):
> >              self.compiler.compiler_so = self.compiler.compiler # no
> -mdll
> >              self.compiler.dll_libraries = [] # no -lmsrvc90
> > +
> >          hv = sys.hexversion
> > -        pythonlib = 'python%d%d' % (hv >> 24, (hv >> 16) & 0xff)
> > +        maj = '%s' % (hv >> 24,)
> > +        min = '%s' % ((hv >> 16) & 0xff,)
> > +
> > +        # Most installs (including the official CPython distribution)
> have a
> > +        # pythonXY.dll somewhere (probably on the PATH).
> > +        libpattern = 'python%s%s'
> > +
> > +        # MinGW puts a libpythonX.Y.dll in the same directory as
> python.exe.
> > +        # If it exists, we use it.
> > +        libpythonpath = os.path.join(os.path.dirname(sys.executable),
> > +                                     'libpython%s.%s.dll' % (maj, min))
> > +        if os.path.exists(libpythonpath):
> > +            libpattern = 'libpython%s.%s'
> > +
> > +        # We don't search for libpythonX.Y.dll on PATH because it is
> valid to
> > +        # run non-MinGW Python from within MinGW and we don't want false
> > +        # positives.
> > +
> > +        pythonlib = libpattern % (maj, min)
> > +        log.info('using %s as Python library name' % pythonlib)
> >          with open('mercurial/hgpythonlib.h', 'wb') as f:
> >              f.write('/* this file is autogenerated by setup.py */\n')
> >              f.write('#define HGPYTHONLIB "%s"\n' % pythonlib)
> >          objects = self.compiler.compile(['mercurial/exewrapper.c'],
> >                                           output_dir=self.build_temp)
> >          dir = os.path.dirname(self.get_ext_fullpath('dummy'))
> >          target = os.path.join(dir, 'hg')
> >          self.compiler.link_executable(objects, target,
>
> Perhaps, we can be even more clever. See for example:
>
> $ python
> Python 2.7.9 (default, Dec 10 2014, 12:28:03) [MSC v.1500 64 bit (AMD64)]
> on win32
> Type "help", "copyright", "credits" or "license" for more information.
> >>> import sys
> >>> sys.dllhandle
> 503316480L
> >>> import ctypes
> >>> _kernel32 = ctypes.windll.kernel32
> >>> size = 600
> >>> buf = ctypes.create_string_buffer(size + 1)
> >>> len = _kernel32.GetModuleFileNameA(sys.dllhandle, ctypes.byref(buf),
> size)
> >>> len
> 32
> >>> buf.value
> 'C:\\Windows\\system32\\python27.dll'
> >>>
>
> Would it be possible to extract the name of the python dll this way?
>
> (see also executablepath in mercurial/win32.py for an example call of
> GetModuleFileNameA)
>
>
I thought about it this morning but was feeling too lazy to figure out how
to turn the integer/handle into a file name. But since you've written the
code, I may plug this into a v3 since it is an even better approach - it
even removes the need for the hex version manipulation! I don't have access
to my Windows machine right now. So it will be at least a few hours...

Patch

diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -364,18 +364,38 @@  class buildhgexe(build_ext):
     description = 'compile hg.exe from mercurial/exewrapper.c'
 
     def build_extensions(self):
         if os.name != 'nt':
             return
         if isinstance(self.compiler, HackedMingw32CCompiler):
             self.compiler.compiler_so = self.compiler.compiler # no -mdll
             self.compiler.dll_libraries = [] # no -lmsrvc90
+
         hv = sys.hexversion
-        pythonlib = 'python%d%d' % (hv >> 24, (hv >> 16) & 0xff)
+        maj = '%s' % (hv >> 24,)
+        min = '%s' % ((hv >> 16) & 0xff,)
+
+        # Most installs (including the official CPython distribution) have a
+        # pythonXY.dll somewhere (probably on the PATH).
+        libpattern = 'python%s%s'
+
+        # MinGW puts a libpythonX.Y.dll in the same directory as python.exe.
+        # If it exists, we use it.
+        libpythonpath = os.path.join(os.path.dirname(sys.executable),
+                                     'libpython%s.%s.dll' % (maj, min))
+        if os.path.exists(libpythonpath):
+            libpattern = 'libpython%s.%s'
+
+        # We don't search for libpythonX.Y.dll on PATH because it is valid to
+        # run non-MinGW Python from within MinGW and we don't want false
+        # positives.
+
+        pythonlib = libpattern % (maj, min)
+        log.info('using %s as Python library name' % pythonlib)
         with open('mercurial/hgpythonlib.h', 'wb') as f:
             f.write('/* this file is autogenerated by setup.py */\n')
             f.write('#define HGPYTHONLIB "%s"\n' % pythonlib)
         objects = self.compiler.compile(['mercurial/exewrapper.c'],
                                          output_dir=self.build_temp)
         dir = os.path.dirname(self.get_ext_fullpath('dummy'))
         target = os.path.join(dir, 'hg')
         self.compiler.link_executable(objects, target,