Patchwork [2,of,2] setup.py: don't rewrite @LIBDIR@ when creating wheels

login
register
mail settings
Submitter Gregory Szorc
Date Dec. 6, 2015, 2:15 a.m.
Message ID <b1cbdb3cf79b4749a05b.1449368104@ubuntu-main>
Download mbox | patch
Permalink /patch/11847/
State Superseded
Delegated to: Yuya Nishihara
Headers show

Comments

Gregory Szorc - Dec. 6, 2015, 2:15 a.m.
# HG changeset patch
# User Gregory Szorc <gregory.szorc@gmail.com>
# Date 1449366770 28800
#      Sat Dec 05 17:52:50 2015 -0800
# Node ID b1cbdb3cf79b4749a05b8a73ca27a2e1dedd90c1
# Parent  6550ffef53c01743d5d0efbd32a13bb9a344579f
setup.py: don't rewrite @LIBDIR@ when creating wheels

This is necessary to produce wheels that install properly. More
details are captured in an in-line comment.

After this patch, produced wheels can be installed via `pip install`
and appear to "just work," including on Windows.
Yuya Nishihara - Dec. 6, 2015, 8:05 a.m.
On Sat, 05 Dec 2015 18:15:04 -0800, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc@gmail.com>
> # Date 1449366770 28800
> #      Sat Dec 05 17:52:50 2015 -0800
> # Node ID b1cbdb3cf79b4749a05b8a73ca27a2e1dedd90c1
> # Parent  6550ffef53c01743d5d0efbd32a13bb9a344579f
> setup.py: don't rewrite @LIBDIR@ when creating wheels
> 
> This is necessary to produce wheels that install properly. More
> details are captured in an in-line comment.
> 
> After this patch, produced wheels can be installed via `pip install`
> and appear to "just work," including on Windows.
> 
> diff --git a/setup.py b/setup.py
> --- a/setup.py
> +++ b/setup.py
> @@ -60,16 +60,17 @@ else:
>          import bz2
>          bz2.BZ2Compressor # silence unused import warning
>      except ImportError:
>          raise SystemExit(
>              "Couldn't import standard bz2 (incomplete Python install).")
>  
>  ispypy = "PyPy" in sys.version
>  
> +import inspect
>  import os, stat, subprocess, time
>  import re
>  import shutil
>  import tempfile
>  from distutils import log
>  if 'FORCE_SETUPTOOLS' in os.environ:
>      from setuptools import setup
>  else:
> @@ -476,16 +477,51 @@ class hginstallscripts(install_scripts):
>      def finalize_options(self):
>          install_scripts.finalize_options(self)
>          self.set_undefined_options('install',
>                                     ('install_lib', 'install_lib'))
>  
>      def run(self):
>          install_scripts.run(self)
>  
> +        # It only makes sense to replace @LIBDIR@ with the install path if
> +        # the install path is known. For wheels, the logic below calculates
> +        # the libdir to be "../..". This is because the internal layout of a
> +        # wheel archive looks like:
> +        #
> +        #   mercurial-3.6.1.data/scripts/hg
> +        #   mercurial/__init__.py
> +        #
> +        # When installing wheels, the subdirectories of the "<pkg>.data"
> +        # directory are translated to system local paths and files therein
> +        # are copied in place. The mercurial/* files are installed into the
> +        # site-packages directory. However, the site-packages directory
> +        # isn't known until wheel install time. This means we have no clue
> +        # at wheel generation time what the installed site-packages directory
> +        # will be. And, wheels don't appear to provide the ability to register
> +        # custom code to run during wheel installation. This all means that
> +        # we can't reliably set the libdir in wheels: the default behavior
> +        # of looking in sys.path must do.
> +        #
> +        # There is no formal API in distutils that exposes the command
> +        # currently being processed. So, we walk the stack. The next best
> +        # alternative is to look at ``self.distribution.commands`` and
> +        # ``self.distribution.have_run``. However, since you can execute
> +        # multiple commands in one setup.py invocation, this isn't reliable.

Another way is to check if the script starts with b'#!python' [1]. If a shebang
line isn't absolute path, that script is highly probably not finalized yet.

 [1]: https://www.python.org/dev/peps/pep-0427/#recommended-installer-features

> +        for frame in inspect.stack():
> +            # We're looking for the method run() of a bdist_wheel class from
> +            # the wheel/bdist_wheel.py file.
> +            code = frame[0].f_code
> +            if code.co_name != 'run':
> +                continue
> +
> +            if inspect.getsourcefile(code).endswith('bdist_wheel.py'):
> +                log.info('not rewriting @LIBDIR@ because building wheel')
> +                return

Uh, hackish. And I got the following error probably because setuptools is
installed as an egg.

Traceback (most recent call last):
  File "setup.py", line 725, in <module>
    **extra)
  File "c:\Python27\lib\distutils\core.py", line 151, in setup
    dist.run_commands()
  File "c:\Python27\lib\distutils\dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "c:\Python27\lib\distutils\dist.py", line 972, in run_command
    cmd_obj.run()
  File "c:\python27\lib\site-packages\wheel-0.26.0-py2.7.egg\wheel\bdist_wheel.py", line 212, in run
    self.run_command('install')
  File "c:\Python27\lib\distutils\cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "c:\Python27\lib\distutils\dist.py", line 972, in run_command
    cmd_obj.run()
  File "build\bdist.win32\egg\setuptools\command\install.py", line 61, in run
  File "c:\Python27\lib\distutils\command\install.py", line 575, in run
    self.run_command(cmd_name)
  File "c:\Python27\lib\distutils\cmd.py", line 326, in run_command
    self.distribution.run_command(command)
  File "c:\Python27\lib\distutils\dist.py", line 972, in run_command
    cmd_obj.run()
  File "setup.py", line 516, in run
    if inspect.getsourcefile(code).endswith('bdist_wheel.py'):
AttributeError: 'NoneType' object has no attribute 'endswith'

Patch

diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -60,16 +60,17 @@  else:
         import bz2
         bz2.BZ2Compressor # silence unused import warning
     except ImportError:
         raise SystemExit(
             "Couldn't import standard bz2 (incomplete Python install).")
 
 ispypy = "PyPy" in sys.version
 
+import inspect
 import os, stat, subprocess, time
 import re
 import shutil
 import tempfile
 from distutils import log
 if 'FORCE_SETUPTOOLS' in os.environ:
     from setuptools import setup
 else:
@@ -476,16 +477,51 @@  class hginstallscripts(install_scripts):
     def finalize_options(self):
         install_scripts.finalize_options(self)
         self.set_undefined_options('install',
                                    ('install_lib', 'install_lib'))
 
     def run(self):
         install_scripts.run(self)
 
+        # It only makes sense to replace @LIBDIR@ with the install path if
+        # the install path is known. For wheels, the logic below calculates
+        # the libdir to be "../..". This is because the internal layout of a
+        # wheel archive looks like:
+        #
+        #   mercurial-3.6.1.data/scripts/hg
+        #   mercurial/__init__.py
+        #
+        # When installing wheels, the subdirectories of the "<pkg>.data"
+        # directory are translated to system local paths and files therein
+        # are copied in place. The mercurial/* files are installed into the
+        # site-packages directory. However, the site-packages directory
+        # isn't known until wheel install time. This means we have no clue
+        # at wheel generation time what the installed site-packages directory
+        # will be. And, wheels don't appear to provide the ability to register
+        # custom code to run during wheel installation. This all means that
+        # we can't reliably set the libdir in wheels: the default behavior
+        # of looking in sys.path must do.
+        #
+        # There is no formal API in distutils that exposes the command
+        # currently being processed. So, we walk the stack. The next best
+        # alternative is to look at ``self.distribution.commands`` and
+        # ``self.distribution.have_run``. However, since you can execute
+        # multiple commands in one setup.py invocation, this isn't reliable.
+        for frame in inspect.stack():
+            # We're looking for the method run() of a bdist_wheel class from
+            # the wheel/bdist_wheel.py file.
+            code = frame[0].f_code
+            if code.co_name != 'run':
+                continue
+
+            if inspect.getsourcefile(code).endswith('bdist_wheel.py'):
+                log.info('not rewriting @LIBDIR@ because building wheel')
+                return
+
         if (os.path.splitdrive(self.install_dir)[0] !=
             os.path.splitdrive(self.install_lib)[0]):
             # can't make relative paths from one drive to another, so use an
             # absolute path instead
             libdir = self.install_lib
         else:
             common = os.path.commonprefix((self.install_dir, self.install_lib))
             rest = self.install_dir[len(common):]