Patchwork [STABLE] packaging: add make target for linux wheels

login
register
mail settings
Submitter Boris Feld
Date April 26, 2017, 5:01 p.m.
Message ID <9605a99df098d4e6e3b0.1493226097@marion-duprey-portable>
Download mbox | patch
Permalink /patch/20301/
State Superseded
Headers show

Comments

Boris Feld - April 26, 2017, 5:01 p.m.
# HG changeset patch
# User Boris Feld <boris.feld@octobus.net>
# Date 1493131801 -7200
#      Tue Apr 25 16:50:01 2017 +0200
# Branch stable
# Node ID 9605a99df098d4e6e3b0943982a21fea6a9421f9
# Parent  6e0368b6e0bb2aa5210daec091c0200583553a78
packaging: add make target for linux wheels

Having linux wheels is going to helps system without compiler or python-dev
plus speed up the installation for everyone.

I followed the manylinux example repository
https://github.com/pypa/python-manylinux-demo
to add a make target (build-linux-wheels) using
official docker image to build python 2 linux wheels
for mercurial. It generates Python 2.6 and Python 2.7 for both
32 and 64 bits architectures.

I had to blacklist several test cases for various reasons:
* test-http-bad-server.t, readline(65537) was transformed in readline(-1)
* test-convert-git.t and test-subrepo-git.t because of the git version
* test-patchbomb-tls.t because of warning using tls 1.0
  It's likely because the docker image is based on centos 5.0 and
  openssl is outdated.
* test-clonebundles.t and test-http-proxy.t because of the following
  output line that changed:
  -  error fetching bundle: (.* refused.*|Protocol not supported) (re)
  +  error fetching bundle: Cannot assign requested address
Sean Farley - April 26, 2017, 6:39 p.m.
Boris Feld <boris.feld@octobus.net> writes:

> # HG changeset patch
> # User Boris Feld <boris.feld@octobus.net>
> # Date 1493131801 -7200
> #      Tue Apr 25 16:50:01 2017 +0200
> # Branch stable
> # Node ID 9605a99df098d4e6e3b0943982a21fea6a9421f9
> # Parent  6e0368b6e0bb2aa5210daec091c0200583553a78
> packaging: add make target for linux wheels
>
> Having linux wheels is going to helps system without compiler or python-dev
> plus speed up the installation for everyone.
>
> I followed the manylinux example repository
> https://github.com/pypa/python-manylinux-demo
> to add a make target (build-linux-wheels) using
> official docker image to build python 2 linux wheels
> for mercurial. It generates Python 2.6 and Python 2.7 for both
> 32 and 64 bits architectures.

Cool!

> I had to blacklist several test cases for various reasons:
> * test-http-bad-server.t, readline(65537) was transformed in readline(-1)
> * test-convert-git.t and test-subrepo-git.t because of the git version
> * test-patchbomb-tls.t because of warning using tls 1.0
>   It's likely because the docker image is based on centos 5.0 and
>   openssl is outdated.
> * test-clonebundles.t and test-http-proxy.t because of the following
>   output line that changed:
>   -  error fetching bundle: (.* refused.*|Protocol not supported) (re)
>   +  error fetching bundle: Cannot assign requested address

I don't know much about wheels, though. Didn't Greg attempt this at one
point?
Augie Fackler - April 27, 2017, 2:17 p.m.
On Wed, Apr 26, 2017 at 07:01:37PM +0200, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld <boris.feld@octobus.net>
> # Date 1493131801 -7200
> #      Tue Apr 25 16:50:01 2017 +0200
> # Branch stable
> # Node ID 9605a99df098d4e6e3b0943982a21fea6a9421f9
> # Parent  6e0368b6e0bb2aa5210daec091c0200583553a78
> packaging: add make target for linux wheels
>
> Having linux wheels is going to helps system without compiler or python-dev
> plus speed up the installation for everyone.
>
> I followed the manylinux example repository
> https://github.com/pypa/python-manylinux-demo
> to add a make target (build-linux-wheels) using
> official docker image to build python 2 linux wheels
> for mercurial. It generates Python 2.6 and Python 2.7 for both
> 32 and 64 bits architectures.
>
> I had to blacklist several test cases for various reasons:
> * test-http-bad-server.t, readline(65537) was transformed in readline(-1)

This is already fixed, so just rebasing your patch should resolve this one. :)

> * test-convert-git.t and test-subrepo-git.t because of the git version
> * test-patchbomb-tls.t because of warning using tls 1.0
>   It's likely because the docker image is based on centos 5.0 and
>   openssl is outdated.

These two make total sense.

> * test-clonebundles.t and test-http-proxy.t because of the following
>   output line that changed:
>   -  error fetching bundle: (.* refused.*|Protocol not supported) (re)
>   +  error fetching bundle: Cannot assign requested address

Sigh. Can you send us a patch that adds the extra case the regex match
for docker? There's no reason these tests shouldn't pass there.

I'd gladly take this for 4.2 and try to add building of linux wheels
to our release discipline, assuming they work correctly. How much did
you test the wheels by hand once installed?
Boris Feld - April 27, 2017, 4:53 p.m.
On 27/04/2017 16:17, Augie Fackler wrote:
> On Wed, Apr 26, 2017 at 07:01:37PM +0200, Boris Feld wrote:
>> # HG changeset patch
>> # User Boris Feld <boris.feld@octobus.net>
>> # Date 1493131801 -7200
>> #      Tue Apr 25 16:50:01 2017 +0200
>> # Branch stable
>> # Node ID 9605a99df098d4e6e3b0943982a21fea6a9421f9
>> # Parent  6e0368b6e0bb2aa5210daec091c0200583553a78
>> packaging: add make target for linux wheels
>>
>> Having linux wheels is going to helps system without compiler or python-dev
>> plus speed up the installation for everyone.
>>
>> I followed the manylinux example repository
>> https://github.com/pypa/python-manylinux-demo
>> to add a make target (build-linux-wheels) using
>> official docker image to build python 2 linux wheels
>> for mercurial. It generates Python 2.6 and Python 2.7 for both
>> 32 and 64 bits architectures.
>>
>> I had to blacklist several test cases for various reasons:
>> * test-http-bad-server.t, readline(65537) was transformed in readline(-1)
> This is already fixed, so just rebasing your patch should resolve this one. :)
It works after rebas, thx.
>
>> * test-convert-git.t and test-subrepo-git.t because of the git version
>> * test-patchbomb-tls.t because of warning using tls 1.0
>>    It's likely because the docker image is based on centos 5.0 and
>>    openssl is outdated.
> These two make total sense.
>
>> * test-clonebundles.t and test-http-proxy.t because of the following
>>    output line that changed:
>>    -  error fetching bundle: (.* refused.*|Protocol not supported) (re)
>>    +  error fetching bundle: Cannot assign requested address
> Sigh. Can you send us a patch that adds the extra case the regex match
> for docker? There's no reason these tests shouldn't pass there.
>
> I'd gladly take this for 4.2 and try to add building of linux wheels
> to our release discipline, assuming they work correctly. How much did
> you test the wheels by hand once installed?
I fixed both tests in the v2 version of my patch.

For testing the generated wheels, they are first installed in the docker 
container after building and tests are runner with the version installed 
from the wheel. I've also tried to install the wheel locally in a fresh 
python 2.7 virtualenv and cloned this repository without bugs. Finally, 
I've installed the wheel in a "Python:2.7" docker container and clone 
the same repository, the output is here: 
https://zerobin.net/?c0f906fb42d32d41#E6Z0KcJnJp26GkJKfYKiDR2SKi2KLJ6xNdwsKgKn/uE=

I'm not sure how much tests we can do to validate the wheels.
Augie Fackler - April 27, 2017, 4:58 p.m.
On Thu, Apr 27, 2017 at 12:53 PM, Boris Feld <boris.feld@octobus.net> wrote:
>
>
> On 27/04/2017 16:17, Augie Fackler wrote:
>
> On Wed, Apr 26, 2017 at 07:01:37PM +0200, Boris Feld wrote:
>
> # HG changeset patch
> # User Boris Feld <boris.feld@octobus.net>
> # Date 1493131801 -7200
> #      Tue Apr 25 16:50:01 2017 +0200
> # Branch stable
> # Node ID 9605a99df098d4e6e3b0943982a21fea6a9421f9
> # Parent  6e0368b6e0bb2aa5210daec091c0200583553a78
> packaging: add make target for linux wheels
>
> Having linux wheels is going to helps system without compiler or python-dev
> plus speed up the installation for everyone.
>
> I followed the manylinux example repository
> https://github.com/pypa/python-manylinux-demo
> to add a make target (build-linux-wheels) using
> official docker image to build python 2 linux wheels
> for mercurial. It generates Python 2.6 and Python 2.7 for both
> 32 and 64 bits architectures.
>
> I had to blacklist several test cases for various reasons:
> * test-http-bad-server.t, readline(65537) was transformed in readline(-1)
>
> This is already fixed, so just rebasing your patch should resolve this one.
> :)
>
> It works after rebas, thx.
>
> * test-convert-git.t and test-subrepo-git.t because of the git version
> * test-patchbomb-tls.t because of warning using tls 1.0
>   It's likely because the docker image is based on centos 5.0 and
>   openssl is outdated.
>
> These two make total sense.
>
> * test-clonebundles.t and test-http-proxy.t because of the following
>   output line that changed:
>   -  error fetching bundle: (.* refused.*|Protocol not supported) (re)
>   +  error fetching bundle: Cannot assign requested address
>
> Sigh. Can you send us a patch that adds the extra case the regex match
> for docker? There's no reason these tests shouldn't pass there.
>
> I'd gladly take this for 4.2 and try to add building of linux wheels
> to our release discipline, assuming they work correctly. How much did
> you test the wheels by hand once installed?
>
> I fixed both tests in the v2 version of my patch.
>
> For testing the generated wheels, they are first installed in the docker
> container after building and tests are runner with the version installed
> from the wheel. I've also tried to install the wheel locally in a fresh
> python 2.7 virtualenv and cloned this repository without bugs. Finally, I've
> installed the wheel in a "Python:2.7" docker container and clone the same
> repository, the output is here:
> https://zerobin.net/?c0f906fb42d32d41#E6Z0KcJnJp26GkJKfYKiDR2SKi2KLJ6xNdwsKgKn/uE=
>
> I'm not sure how much tests we can do to validate the wheels.

The only other things I can think of that we should test are:

1) Does `hg help` seem to work?
2) Does an extension work (maybe try children since it's a trivial extension?)
3) Do localizations seem to work?

Patch

diff -r 6e0368b6e0bb -r 9605a99df098 .hgignore
--- a/.hgignore	Thu Apr 20 22:51:28 2017 +0900
+++ b/.hgignore	Tue Apr 25 16:50:01 2017 +0200
@@ -62,6 +62,9 @@ 
 mercurial/osutil.py
 mercurial/parsers.py
 
+# Generated wheels
+wheelhouse/
+
 syntax: regexp
 ^\.pc/
 ^\.(pydev)?project
diff -r 6e0368b6e0bb -r 9605a99df098 Makefile
--- a/Makefile	Thu Apr 20 22:51:28 2017 +0900
+++ b/Makefile	Tue Apr 25 16:50:01 2017 +0200
@@ -270,6 +270,14 @@ 
 	mkdir -p packages/centos7
 	contrib/dockerrpm centos7
 
+linux-wheels: linux-wheels-x86_64 linux-wheels-i686
+	
+linux-wheels-x86_64:
+	docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src quay.io/pypa/manylinux1_x86_64 /src/contrib/build-linux-wheels.sh
+	
+linux-wheels-i686:
+	docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src quay.io/pypa/manylinux1_i686 linux32 /src/contrib/build-linux-wheels.sh
+
 .PHONY: help all local build doc cleanbutpackages clean install install-bin \
 	install-doc install-home install-home-bin install-home-doc \
 	dist dist-notests check tests check-code update-pot \
@@ -278,4 +286,5 @@ 
 	docker-ubuntu-xenial docker-ubuntu-xenial-ppa \
 	docker-ubuntu-yakkety docker-ubuntu-yakkety-ppa \
 	fedora20 docker-fedora20 fedora21 docker-fedora21 \
-	centos5 docker-centos5 centos6 docker-centos6 centos7 docker-centos7
+	centos5 docker-centos5 centos6 docker-centos6 centos7 docker-centos7 \
+	linux-wheels
diff -r 6e0368b6e0bb -r 9605a99df098 contrib/build-linux-wheels.sh
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/contrib/build-linux-wheels.sh	Tue Apr 25 16:50:01 2017 +0200
@@ -0,0 +1,34 @@ 
+#!/bin/bash
+# This file is directly inspired by
+# https://github.com/pypa/python-manylinux-demo/blob/master/travis/build-wheels.sh
+set -e -x
+
+PYTHON_TARGETS=$(ls -d /opt/python/cp2*/bin)
+
+# Create an user for the tests
+useradd hgbuilder
+
+# Bypass uid/gid problems
+cp -R /src /io && chown -R hgbuilder:hgbuilder /io
+
+# Compile wheels for Python 2.X
+for PYBIN in $PYTHON_TARGETS; do
+    "${PYBIN}/pip" wheel /io/ -w wheelhouse/
+done
+
+# Bundle external shared libraries into the wheels with
+# auditwheel (https://github.com/pypa/auditwheel) repair.
+# It also fix the ABI tag on the wheel making it pip installable.
+for whl in wheelhouse/*.whl; do
+    auditwheel repair "$whl" -w /src/wheelhouse/
+done
+
+# Install packages and run the tests for all Python versions
+cd /io/tests/
+
+for PYBIN in $PYTHON_TARGETS; do
+    # Install mercurial wheel as root
+    "${PYBIN}/pip" install mercurial --no-index -f /src/wheelhouse
+    # But run tests as hgbuilder user (non-root)
+    su hgbuilder -c "\"${PYBIN}/python\" /io/tests/run-tests.py --with-hg=\"${PYBIN}/hg\" --blacklist=/io/contrib/linux-wheel-centos5-blacklist"
+done
diff -r 6e0368b6e0bb -r 9605a99df098 contrib/linux-wheel-centos5-blacklist
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/contrib/linux-wheel-centos5-blacklist	Tue Apr 25 16:50:01 2017 +0200
@@ -0,0 +1,6 @@ 
+test-http-bad-server.t
+test-convert-git.t
+test-subrepo-git.t
+test-patchbomb-tls.t
+test-clonebundles.t
+test-http-proxy.t