Patchwork [STABLE,V2] packaging: add make target for linux wheels

login
register
mail settings
Submitter Boris Feld
Date April 27, 2017, 4:49 p.m.
Message ID <d49b79b313989b62482d.1493311790@marion-duprey-portable>
Download mbox | patch
Permalink /patch/20304/
State Accepted
Headers show

Comments

Boris Feld - April 27, 2017, 4:49 p.m.
# HG changeset patch
# User Boris Feld <boris.feld@octobus.net>
# Date 1493131801 -7200
#      Tue Apr 25 16:50:01 2017 +0200
# Branch stable
# Node ID d49b79b313989b62482da8e560fafa3373ae90b5
# Parent  52cca17ac523b490acdab259e794e7c2898efeda
packaging: add make target for linux wheels

Having linux wheels is going to helps system without compiler or python-dev
plus speed up the installation for everyone.

I followed the manylinux example repository
https://github.com/pypa/python-manylinux-demo
to add a make target (build-linux-wheels) using
official docker image to build python 2 linux wheels
for mercurial. It generates Python 2.6 and Python 2.7 for both
32 and 64 bits architectures.

I had to blacklist several test cases for various reasons:
* test-convert-git.t and test-subrepo-git.t because of the git version
* test-patchbomb-tls.t because of warning using tls 1.0
  It's likely because the docker image is based on centos 5.0 and
  openssl is outdated.
Augie Fackler - April 28, 2017, 2:51 p.m.
On Thu, Apr 27, 2017 at 06:49:50PM +0200, Boris Feld wrote:
> # HG changeset patch
> # User Boris Feld <boris.feld@octobus.net>
> # Date 1493131801 -7200
> #      Tue Apr 25 16:50:01 2017 +0200
> # Branch stable
> # Node ID d49b79b313989b62482da8e560fafa3373ae90b5
> # Parent  52cca17ac523b490acdab259e794e7c2898efeda
> packaging: add make target for linux wheels

In general very good - I've got a few questions throughout the patch,
but we're very close to something I can land.

> diff -r 52cca17ac523 -r d49b79b31398 Makefile
> --- a/Makefile	Mon Apr 24 14:38:30 2017 -0400
> +++ b/Makefile	Tue Apr 25 16:50:01 2017 +0200
> @@ -270,6 +270,14 @@
>       mkdir -p packages/centos7
>       contrib/dockerrpm centos7
>
> +linux-wheels: linux-wheels-x86_64 linux-wheels-i686
> +
> +linux-wheels-x86_64:
> +	docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src quay.io/pypa/manylinux1_x86_64 /src/contrib/build-linux-wheels.sh
> +
> +linux-wheels-i686:
> +	docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src quay.io/pypa/manylinux1_i686 linux32 /src/contrib/build-linux-wheels.sh

In my attempts to test this, I had to explicitly use bash
/src/contrib/build-linux-wheels.sh - does that make any sense?

> +
>  .PHONY: help all local build doc cleanbutpackages clean install install-bin \
>       install-doc install-home install-home-bin install-home-doc \
>       dist dist-notests check tests check-code update-pot \
> @@ -278,4 +286,5 @@
>       docker-ubuntu-xenial docker-ubuntu-xenial-ppa \
>       docker-ubuntu-yakkety docker-ubuntu-yakkety-ppa \
>       fedora20 docker-fedora20 fedora21 docker-fedora21 \
> -	centos5 docker-centos5 centos6 docker-centos6 centos7 docker-centos7
> +	centos5 docker-centos5 centos6 docker-centos6 centos7 docker-centos7 \
> +	linux-wheels

Should the linux-wheels-{x86_64,i686} targets be in phony as well?

> diff -r 52cca17ac523 -r d49b79b31398 contrib/build-linux-wheels.sh
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/contrib/build-linux-wheels.sh	Tue Apr 25 16:50:01 2017 +0200

(script looks fine)

> diff -r 52cca17ac523 -r d49b79b31398 contrib/linux-wheel-centos5-blacklist
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/contrib/linux-wheel-centos5-blacklist	Tue Apr 25 16:50:01 2017 +0200
> @@ -0,0 +1,3 @@
> +test-convert-git.t
> +test-subrepo-git.t
> +test-patchbomb-tls.t
> diff -r 52cca17ac523 -r d49b79b31398 tests/test-clonebundles.t
> --- a/tests/test-clonebundles.t	Mon Apr 24 14:38:30 2017 -0400
> +++ b/tests/test-clonebundles.t	Tue Apr 25 16:50:01 2017 +0200
> @@ -61,7 +61,7 @@
>    $ echo "http://localhost:$HGPORT1/bundle.hg" > server/.hg/clonebundles.manifest
>    $ hg clone http://localhost:$HGPORT server-not-runner
>    applying clone bundle from http://localhost:$HGPORT1/bundle.hg
> -  error fetching bundle: (.* refused.*|Protocol not supported) (re)
> +  error fetching bundle: (.* refused.*|Protocol not supported|Cannot assign requested address) (re)
>    abort: error applying bundle
>    (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
>    [255]
> diff -r 52cca17ac523 -r d49b79b31398 tests/test-http-proxy.t
> --- a/tests/test-http-proxy.t	Mon Apr 24 14:38:30 2017 -0400
> +++ b/tests/test-http-proxy.t	Tue Apr 25 16:50:01 2017 +0200
> @@ -87,7 +87,7 @@
>  misconfigured hosts)
>
>    $ http_proxy=localhost:$HGPORT2 hg clone --config http_proxy.always=True http://localhost:$HGPORT/ f
> -  abort: error: (Connection refused|Protocol not supported|.* actively refused it) (re)
> +  abort: error: (Connection refused|Protocol not supported|.* actively refused it|Cannot assign requested address) (re)
>    [255]
>
>  do not use the proxy if it is in the no list

I've split these two test changes into their own commit, with this
message (but still your name):
"""
tests: fix two http tests to also pass inside manylinux1 docker

This will let us build Linux wheels.
"""

Do you mind if I push that?

> diff -r 52cca17ac523 -r d49b79b31398 tests/test-http-bad-server.t
> --- a/tests/test-http-bad-server.t	Mon Apr 24 14:38:30 2017 -0400
> +++ b/tests/test-http-bad-server.t	Tue Apr 25 16:50:01 2017 +0200
> @@ -36,6 +36,7 @@
>    $ hg clone http://localhost:$HGPORT/ clone
>    abort: error: Connection reset by peer (no-windows !)
>    abort: error: An existing connection was forcibly closed by the remote host (windows !)
> +  (?)

Can you elaborate here on what this change is about? It looks very weird.

>    [255]
>
>  (The server exits on its own, but there is a race between that and starting a new server.
> @@ -223,7 +224,7 @@
>    readline(92 from -1) -> (41) content-type: application/mercurial-0.1\r\n
>    readline(51 from -1) -> (19) vary: X-HgProto-1\r\n
>    readline(32 from -1) -> (19) x-hgargs-post: 28\r\n
> -  readline(13 from -1) -> (13) x-hgproto-1:
> +  readline(13 from -1) -> (13) x-hgproto-1:

did you mean to remove this trailing whitespace?

>    read limit reached; closing socket
>    readline\(315 from (-1|65537)\) -> \(27\) POST /\?cmd=batch HTTP/1.1\\r\\n (re)
>    readline(288 from -1) -> (27) Accept-Encoding: identity\r\n
> diff -r 52cca17ac523 -r d49b79b31398 tests/test-worker.t
> --- a/tests/test-worker.t	Mon Apr 24 14:38:30 2017 -0400
> +++ b/tests/test-worker.t	Tue Apr 25 16:50:01 2017 +0200
> @@ -78,6 +78,7 @@
>    > test 100000.0 abort
>    start
>    abort: known exception
> +  (?)

Same question as the other place this happens.

>    [255]
>
>    $ hg --config "extensions.t=$abspath" --config 'worker.numcpus=2' \
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Boris Feld - April 28, 2017, 3:36 p.m.
On 28/04/2017 16:51, Augie Fackler wrote:
> On Thu, Apr 27, 2017 at 06:49:50PM +0200, Boris Feld wrote:
>> # HG changeset patch
>> # User Boris Feld <boris.feld@octobus.net>
>> # Date 1493131801 -7200
>> #      Tue Apr 25 16:50:01 2017 +0200
>> # Branch stable
>> # Node ID d49b79b313989b62482da8e560fafa3373ae90b5
>> # Parent  52cca17ac523b490acdab259e794e7c2898efeda
>> packaging: add make target for linux wheels
> In general very good - I've got a few questions throughout the patch,
> but we're very close to something I can land.
>
>> diff -r 52cca17ac523 -r d49b79b31398 Makefile
>> --- a/Makefile	Mon Apr 24 14:38:30 2017 -0400
>> +++ b/Makefile	Tue Apr 25 16:50:01 2017 +0200
>> @@ -270,6 +270,14 @@
>>        mkdir -p packages/centos7
>>        contrib/dockerrpm centos7
>>
>> +linux-wheels: linux-wheels-x86_64 linux-wheels-i686
>> +
>> +linux-wheels-x86_64:
>> +	docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src quay.io/pypa/manylinux1_x86_64 /src/contrib/build-linux-wheels.sh
>> +
>> +linux-wheels-i686:
>> +	docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src quay.io/pypa/manylinux1_i686 linux32 /src/contrib/build-linux-wheels.sh
> In my attempts to test this, I had to explicitly use bash
> /src/contrib/build-linux-wheels.sh - does that make any sense?
I'm not sure to understand why you mean? Did you had to add bash before 
the script name in the Makefile ? Or when you wanted to test by hand?

What is your version of docker? You can find out with 'docker --version'.
>
>> +
>>   .PHONY: help all local build doc cleanbutpackages clean install install-bin \
>>        install-doc install-home install-home-bin install-home-doc \
>>        dist dist-notests check tests check-code update-pot \
>> @@ -278,4 +286,5 @@
>>        docker-ubuntu-xenial docker-ubuntu-xenial-ppa \
>>        docker-ubuntu-yakkety docker-ubuntu-yakkety-ppa \
>>        fedora20 docker-fedora20 fedora21 docker-fedora21 \
>> -	centos5 docker-centos5 centos6 docker-centos6 centos7 docker-centos7
>> +	centos5 docker-centos5 centos6 docker-centos6 centos7 docker-centos7 \
>> +	linux-wheels
> Should the linux-wheels-{x86_64,i686} targets be in phony as well?
I don't have a strong opinion, I'm not sure if people will want to 
launch only one of them.
>
>> diff -r 52cca17ac523 -r d49b79b31398 contrib/build-linux-wheels.sh
>> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>> +++ b/contrib/build-linux-wheels.sh	Tue Apr 25 16:50:01 2017 +0200
> (script looks fine)
>
>> diff -r 52cca17ac523 -r d49b79b31398 contrib/linux-wheel-centos5-blacklist
>> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
>> +++ b/contrib/linux-wheel-centos5-blacklist	Tue Apr 25 16:50:01 2017 +0200
>> @@ -0,0 +1,3 @@
>> +test-convert-git.t
>> +test-subrepo-git.t
>> +test-patchbomb-tls.t
>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-clonebundles.t
>> --- a/tests/test-clonebundles.t	Mon Apr 24 14:38:30 2017 -0400
>> +++ b/tests/test-clonebundles.t	Tue Apr 25 16:50:01 2017 +0200
>> @@ -61,7 +61,7 @@
>>     $ echo "http://localhost:$HGPORT1/bundle.hg" > server/.hg/clonebundles.manifest
>>     $ hg clone http://localhost:$HGPORT server-not-runner
>>     applying clone bundle from http://localhost:$HGPORT1/bundle.hg
>> -  error fetching bundle: (.* refused.*|Protocol not supported) (re)
>> +  error fetching bundle: (.* refused.*|Protocol not supported|Cannot assign requested address) (re)
>>     abort: error applying bundle
>>     (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
>>     [255]
>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-http-proxy.t
>> --- a/tests/test-http-proxy.t	Mon Apr 24 14:38:30 2017 -0400
>> +++ b/tests/test-http-proxy.t	Tue Apr 25 16:50:01 2017 +0200
>> @@ -87,7 +87,7 @@
>>   misconfigured hosts)
>>
>>     $ http_proxy=localhost:$HGPORT2 hg clone --config http_proxy.always=True http://localhost:$HGPORT/ f
>> -  abort: error: (Connection refused|Protocol not supported|.* actively refused it) (re)
>> +  abort: error: (Connection refused|Protocol not supported|.* actively refused it|Cannot assign requested address) (re)
>>     [255]
>>
>>   do not use the proxy if it is in the no list
> I've split these two test changes into their own commit, with this
> message (but still your name):
> """
> tests: fix two http tests to also pass inside manylinux1 docker
>
> This will let us build Linux wheels.
> """
>
> Do you mind if I push that?
Not at all! Thank you for doing it, I should have done it myself.
>
>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-http-bad-server.t
>> --- a/tests/test-http-bad-server.t	Mon Apr 24 14:38:30 2017 -0400
>> +++ b/tests/test-http-bad-server.t	Tue Apr 25 16:50:01 2017 +0200
>> @@ -36,6 +36,7 @@
>>     $ hg clone http://localhost:$HGPORT/ clone
>>     abort: error: Connection reset by peer (no-windows !)
>>     abort: error: An existing connection was forcibly closed by the remote host (windows !)
>> +  (?)
> Can you elaborate here on what this change is about? It looks very weird.
I randomly get a "abort: error: Cannot assign requested address" and I 
tried to get rid of it with "(?)" without luck for now.
>
>>     [255]
>>
>>   (The server exits on its own, but there is a race between that and starting a new server.
>> @@ -223,7 +224,7 @@
>>     readline(92 from -1) -> (41) content-type: application/mercurial-0.1\r\n
>>     readline(51 from -1) -> (19) vary: X-HgProto-1\r\n
>>     readline(32 from -1) -> (19) x-hgargs-post: 28\r\n
>> -  readline(13 from -1) -> (13) x-hgproto-1:
>> +  readline(13 from -1) -> (13) x-hgproto-1:
> did you mean to remove this trailing whitespace?
No, it's a mistake.
>
>>     read limit reached; closing socket
>>     readline\(315 from (-1|65537)\) -> \(27\) POST /\?cmd=batch HTTP/1.1\\r\\n (re)
>>     readline(288 from -1) -> (27) Accept-Encoding: identity\r\n
>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-worker.t
>> --- a/tests/test-worker.t	Mon Apr 24 14:38:30 2017 -0400
>> +++ b/tests/test-worker.t	Tue Apr 25 16:50:01 2017 +0200
>> @@ -78,6 +78,7 @@
>>     > test 100000.0 abort
>>     start
>>     abort: known exception
>> +  (?)
> Same question as the other place this happens.
Same problem, I randomly get the "abort: error: Cannot assign requested 
address" message.
>
>>     [255]
>>
>>     $ hg --config "extensions.t=$abspath" --config 'worker.numcpus=2' \
>> _______________________________________________
>> Mercurial-devel mailing list
>> Mercurial-devel@mercurial-scm.org
>> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Augie Fackler - April 28, 2017, 3:43 p.m.
On Fri, Apr 28, 2017 at 11:36 AM, lothiraldan <boris.feld@octobus.net> wrote:
> On 28/04/2017 16:51, Augie Fackler wrote:
>>
>> On Thu, Apr 27, 2017 at 06:49:50PM +0200, Boris Feld wrote:
>>>
>>> # HG changeset patch
>>> # User Boris Feld <boris.feld@octobus.net>
>>> # Date 1493131801 -7200
>>> #      Tue Apr 25 16:50:01 2017 +0200
>>> # Branch stable
>>> # Node ID d49b79b313989b62482da8e560fafa3373ae90b5
>>> # Parent  52cca17ac523b490acdab259e794e7c2898efeda
>>> packaging: add make target for linux wheels
>>
>> In general very good - I've got a few questions throughout the patch,
>> but we're very close to something I can land.
>>
>>> diff -r 52cca17ac523 -r d49b79b31398 Makefile
>>> --- a/Makefile  Mon Apr 24 14:38:30 2017 -0400
>>> +++ b/Makefile  Tue Apr 25 16:50:01 2017 +0200
>>> @@ -270,6 +270,14 @@
>>>        mkdir -p packages/centos7
>>>        contrib/dockerrpm centos7
>>>
>>> +linux-wheels: linux-wheels-x86_64 linux-wheels-i686
>>> +
>>> +linux-wheels-x86_64:
>>> +       docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src
>>> quay.io/pypa/manylinux1_x86_64 /src/contrib/build-linux-wheels.sh
>>> +
>>> +linux-wheels-i686:
>>> +       docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src
>>> quay.io/pypa/manylinux1_i686 linux32 /src/contrib/build-linux-wheels.sh
>>
>> In my attempts to test this, I had to explicitly use bash
>> /src/contrib/build-linux-wheels.sh - does that make any sense?
>
> I'm not sure to understand why you mean? Did you had to add bash before the
> script name in the Makefile ? Or when you wanted to test by hand?

I had to add it in the Makefile.

>
> What is your version of docker? You can find out with 'docker --version'.

Docker version 1.7.1, build 786b29d

but if adding bash to the command line also works for you that'd be my
preference, since it works on more systems (this is the docker you get
from Ubuntu 14.04)

>>
>>
>>> +
>>>   .PHONY: help all local build doc cleanbutpackages clean install
>>> install-bin \
>>>        install-doc install-home install-home-bin install-home-doc \
>>>        dist dist-notests check tests check-code update-pot \
>>> @@ -278,4 +286,5 @@
>>>        docker-ubuntu-xenial docker-ubuntu-xenial-ppa \
>>>        docker-ubuntu-yakkety docker-ubuntu-yakkety-ppa \
>>>        fedora20 docker-fedora20 fedora21 docker-fedora21 \
>>> -       centos5 docker-centos5 centos6 docker-centos6 centos7
>>> docker-centos7
>>> +       centos5 docker-centos5 centos6 docker-centos6 centos7
>>> docker-centos7 \
>>> +       linux-wheels
>>
>> Should the linux-wheels-{x86_64,i686} targets be in phony as well?
>
> I don't have a strong opinion, I'm not sure if people will want to launch
> only one of them.
>
>>
>>> diff -r 52cca17ac523 -r d49b79b31398 contrib/build-linux-wheels.sh
>>> --- /dev/null   Thu Jan 01 00:00:00 1970 +0000
>>> +++ b/contrib/build-linux-wheels.sh     Tue Apr 25 16:50:01 2017 +0200
>>
>> (script looks fine)
>>
>>> diff -r 52cca17ac523 -r d49b79b31398
>>> contrib/linux-wheel-centos5-blacklist
>>> --- /dev/null   Thu Jan 01 00:00:00 1970 +0000
>>> +++ b/contrib/linux-wheel-centos5-blacklist     Tue Apr 25 16:50:01 2017
>>> +0200
>>> @@ -0,0 +1,3 @@
>>> +test-convert-git.t
>>> +test-subrepo-git.t
>>> +test-patchbomb-tls.t
>>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-clonebundles.t
>>> --- a/tests/test-clonebundles.t Mon Apr 24 14:38:30 2017 -0400
>>> +++ b/tests/test-clonebundles.t Tue Apr 25 16:50:01 2017 +0200
>>> @@ -61,7 +61,7 @@
>>>     $ echo "http://localhost:$HGPORT1/bundle.hg" >
>>> server/.hg/clonebundles.manifest
>>>     $ hg clone http://localhost:$HGPORT server-not-runner
>>>     applying clone bundle from http://localhost:$HGPORT1/bundle.hg
>>> -  error fetching bundle: (.* refused.*|Protocol not supported) (re)
>>> +  error fetching bundle: (.* refused.*|Protocol not supported|Cannot
>>> assign requested address) (re)
>>>     abort: error applying bundle
>>>     (if this error persists, consider contacting the server operator or
>>> disable clone bundles via "--config ui.clonebundles=false")
>>>     [255]
>>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-http-proxy.t
>>> --- a/tests/test-http-proxy.t   Mon Apr 24 14:38:30 2017 -0400
>>> +++ b/tests/test-http-proxy.t   Tue Apr 25 16:50:01 2017 +0200
>>> @@ -87,7 +87,7 @@
>>>   misconfigured hosts)
>>>
>>>     $ http_proxy=localhost:$HGPORT2 hg clone --config
>>> http_proxy.always=True http://localhost:$HGPORT/ f
>>> -  abort: error: (Connection refused|Protocol not supported|.* actively
>>> refused it) (re)
>>> +  abort: error: (Connection refused|Protocol not supported|.* actively
>>> refused it|Cannot assign requested address) (re)
>>>     [255]
>>>
>>>   do not use the proxy if it is in the no list
>>
>> I've split these two test changes into their own commit, with this
>> message (but still your name):
>> """
>> tests: fix two http tests to also pass inside manylinux1 docker
>>
>> This will let us build Linux wheels.
>> """
>>
>> Do you mind if I push that?
>
> Not at all! Thank you for doing it, I should have done it myself.
>>
>>
>>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-http-bad-server.t
>>> --- a/tests/test-http-bad-server.t      Mon Apr 24 14:38:30 2017 -0400
>>> +++ b/tests/test-http-bad-server.t      Tue Apr 25 16:50:01 2017 +0200
>>> @@ -36,6 +36,7 @@
>>>     $ hg clone http://localhost:$HGPORT/ clone
>>>     abort: error: Connection reset by peer (no-windows !)
>>>     abort: error: An existing connection was forcibly closed by the
>>> remote host (windows !)
>>> +  (?)
>>
>> Can you elaborate here on what this change is about? It looks very weird.
>
> I randomly get a "abort: error: Cannot assign requested address" and I tried
> to get rid of it with "(?)" without luck for now.

Hm, that's odd.

>>
>>
>>>     [255]
>>>
>>>   (The server exits on its own, but there is a race between that and
>>> starting a new server.
>>> @@ -223,7 +224,7 @@
>>>     readline(92 from -1) -> (41) content-type:
>>> application/mercurial-0.1\r\n
>>>     readline(51 from -1) -> (19) vary: X-HgProto-1\r\n
>>>     readline(32 from -1) -> (19) x-hgargs-post: 28\r\n
>>> -  readline(13 from -1) -> (13) x-hgproto-1:
>>> +  readline(13 from -1) -> (13) x-hgproto-1:
>>
>> did you mean to remove this trailing whitespace?
>
> No, it's a mistake.
>>
>>
>>>     read limit reached; closing socket
>>>     readline\(315 from (-1|65537)\) -> \(27\) POST /\?cmd=batch
>>> HTTP/1.1\\r\\n (re)
>>>     readline(288 from -1) -> (27) Accept-Encoding: identity\r\n
>>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-worker.t
>>> --- a/tests/test-worker.t       Mon Apr 24 14:38:30 2017 -0400
>>> +++ b/tests/test-worker.t       Tue Apr 25 16:50:01 2017 +0200
>>> @@ -78,6 +78,7 @@
>>>     > test 100000.0 abort
>>>     start
>>>     abort: known exception
>>> +  (?)
>>
>> Same question as the other place this happens.
>
> Same problem, I randomly get the "abort: error: Cannot assign requested
> address" message.
>
>>
>>>     [255]
>>>
>>>     $ hg --config "extensions.t=$abspath" --config 'worker.numcpus=2' \
>>> _______________________________________________
>>> Mercurial-devel mailing list
>>> Mercurial-devel@mercurial-scm.org
>>> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>
>
Boris Feld - April 28, 2017, 4:34 p.m.
On 28/04/2017 17:43, Augie Fackler wrote:
> On Fri, Apr 28, 2017 at 11:36 AM, lothiraldan <boris.feld@octobus.net> wrote:
>> On 28/04/2017 16:51, Augie Fackler wrote:
>>> On Thu, Apr 27, 2017 at 06:49:50PM +0200, Boris Feld wrote:
>>>> # HG changeset patch
>>>> # User Boris Feld <boris.feld@octobus.net>
>>>> # Date 1493131801 -7200
>>>> #      Tue Apr 25 16:50:01 2017 +0200
>>>> # Branch stable
>>>> # Node ID d49b79b313989b62482da8e560fafa3373ae90b5
>>>> # Parent  52cca17ac523b490acdab259e794e7c2898efeda
>>>> packaging: add make target for linux wheels
>>> In general very good - I've got a few questions throughout the patch,
>>> but we're very close to something I can land.
>>>
>>>> diff -r 52cca17ac523 -r d49b79b31398 Makefile
>>>> --- a/Makefile  Mon Apr 24 14:38:30 2017 -0400
>>>> +++ b/Makefile  Tue Apr 25 16:50:01 2017 +0200
>>>> @@ -270,6 +270,14 @@
>>>>         mkdir -p packages/centos7
>>>>         contrib/dockerrpm centos7
>>>>
>>>> +linux-wheels: linux-wheels-x86_64 linux-wheels-i686
>>>> +
>>>> +linux-wheels-x86_64:
>>>> +       docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src
>>>> quay.io/pypa/manylinux1_x86_64 /src/contrib/build-linux-wheels.sh
>>>> +
>>>> +linux-wheels-i686:
>>>> +       docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src
>>>> quay.io/pypa/manylinux1_i686 linux32 /src/contrib/build-linux-wheels.sh
>>> In my attempts to test this, I had to explicitly use bash
>>> /src/contrib/build-linux-wheels.sh - does that make any sense?
>> I'm not sure to understand why you mean? Did you had to add bash before the
>> script name in the Makefile ? Or when you wanted to test by hand?
> I had to add it in the Makefile.
>
>> What is your version of docker? You can find out with 'docker --version'.
> Docker version 1.7.1, build 786b29d
>
> but if adding bash to the command line also works for you that'd be my
> preference, since it works on more systems (this is the docker you get
> from Ubuntu 14.04)
That works for me! Do you want me to send a new patch?
>
>>>
>>>> +
>>>>    .PHONY: help all local build doc cleanbutpackages clean install
>>>> install-bin \
>>>>         install-doc install-home install-home-bin install-home-doc \
>>>>         dist dist-notests check tests check-code update-pot \
>>>> @@ -278,4 +286,5 @@
>>>>         docker-ubuntu-xenial docker-ubuntu-xenial-ppa \
>>>>         docker-ubuntu-yakkety docker-ubuntu-yakkety-ppa \
>>>>         fedora20 docker-fedora20 fedora21 docker-fedora21 \
>>>> -       centos5 docker-centos5 centos6 docker-centos6 centos7
>>>> docker-centos7
>>>> +       centos5 docker-centos5 centos6 docker-centos6 centos7
>>>> docker-centos7 \
>>>> +       linux-wheels
>>> Should the linux-wheels-{x86_64,i686} targets be in phony as well?
>> I don't have a strong opinion, I'm not sure if people will want to launch
>> only one of them.
>>
>>>> diff -r 52cca17ac523 -r d49b79b31398 contrib/build-linux-wheels.sh
>>>> --- /dev/null   Thu Jan 01 00:00:00 1970 +0000
>>>> +++ b/contrib/build-linux-wheels.sh     Tue Apr 25 16:50:01 2017 +0200
>>> (script looks fine)
>>>
>>>> diff -r 52cca17ac523 -r d49b79b31398
>>>> contrib/linux-wheel-centos5-blacklist
>>>> --- /dev/null   Thu Jan 01 00:00:00 1970 +0000
>>>> +++ b/contrib/linux-wheel-centos5-blacklist     Tue Apr 25 16:50:01 2017
>>>> +0200
>>>> @@ -0,0 +1,3 @@
>>>> +test-convert-git.t
>>>> +test-subrepo-git.t
>>>> +test-patchbomb-tls.t
>>>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-clonebundles.t
>>>> --- a/tests/test-clonebundles.t Mon Apr 24 14:38:30 2017 -0400
>>>> +++ b/tests/test-clonebundles.t Tue Apr 25 16:50:01 2017 +0200
>>>> @@ -61,7 +61,7 @@
>>>>      $ echo "http://localhost:$HGPORT1/bundle.hg" >
>>>> server/.hg/clonebundles.manifest
>>>>      $ hg clone http://localhost:$HGPORT server-not-runner
>>>>      applying clone bundle from http://localhost:$HGPORT1/bundle.hg
>>>> -  error fetching bundle: (.* refused.*|Protocol not supported) (re)
>>>> +  error fetching bundle: (.* refused.*|Protocol not supported|Cannot
>>>> assign requested address) (re)
>>>>      abort: error applying bundle
>>>>      (if this error persists, consider contacting the server operator or
>>>> disable clone bundles via "--config ui.clonebundles=false")
>>>>      [255]
>>>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-http-proxy.t
>>>> --- a/tests/test-http-proxy.t   Mon Apr 24 14:38:30 2017 -0400
>>>> +++ b/tests/test-http-proxy.t   Tue Apr 25 16:50:01 2017 +0200
>>>> @@ -87,7 +87,7 @@
>>>>    misconfigured hosts)
>>>>
>>>>      $ http_proxy=localhost:$HGPORT2 hg clone --config
>>>> http_proxy.always=True http://localhost:$HGPORT/ f
>>>> -  abort: error: (Connection refused|Protocol not supported|.* actively
>>>> refused it) (re)
>>>> +  abort: error: (Connection refused|Protocol not supported|.* actively
>>>> refused it|Cannot assign requested address) (re)
>>>>      [255]
>>>>
>>>>    do not use the proxy if it is in the no list
>>> I've split these two test changes into their own commit, with this
>>> message (but still your name):
>>> """
>>> tests: fix two http tests to also pass inside manylinux1 docker
>>>
>>> This will let us build Linux wheels.
>>> """
>>>
>>> Do you mind if I push that?
>> Not at all! Thank you for doing it, I should have done it myself.
>>>
>>>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-http-bad-server.t
>>>> --- a/tests/test-http-bad-server.t      Mon Apr 24 14:38:30 2017 -0400
>>>> +++ b/tests/test-http-bad-server.t      Tue Apr 25 16:50:01 2017 +0200
>>>> @@ -36,6 +36,7 @@
>>>>      $ hg clone http://localhost:$HGPORT/ clone
>>>>      abort: error: Connection reset by peer (no-windows !)
>>>>      abort: error: An existing connection was forcibly closed by the
>>>> remote host (windows !)
>>>> +  (?)
>>> Can you elaborate here on what this change is about? It looks very weird.
>> I randomly get a "abort: error: Cannot assign requested address" and I tried
>> to get rid of it with "(?)" without luck for now.
> Hm, that's odd.
>
>>>
>>>>      [255]
>>>>
>>>>    (The server exits on its own, but there is a race between that and
>>>> starting a new server.
>>>> @@ -223,7 +224,7 @@
>>>>      readline(92 from -1) -> (41) content-type:
>>>> application/mercurial-0.1\r\n
>>>>      readline(51 from -1) -> (19) vary: X-HgProto-1\r\n
>>>>      readline(32 from -1) -> (19) x-hgargs-post: 28\r\n
>>>> -  readline(13 from -1) -> (13) x-hgproto-1:
>>>> +  readline(13 from -1) -> (13) x-hgproto-1:
>>> did you mean to remove this trailing whitespace?
>> No, it's a mistake.
>>>
>>>>      read limit reached; closing socket
>>>>      readline\(315 from (-1|65537)\) -> \(27\) POST /\?cmd=batch
>>>> HTTP/1.1\\r\\n (re)
>>>>      readline(288 from -1) -> (27) Accept-Encoding: identity\r\n
>>>> diff -r 52cca17ac523 -r d49b79b31398 tests/test-worker.t
>>>> --- a/tests/test-worker.t       Mon Apr 24 14:38:30 2017 -0400
>>>> +++ b/tests/test-worker.t       Tue Apr 25 16:50:01 2017 +0200
>>>> @@ -78,6 +78,7 @@
>>>>      > test 100000.0 abort
>>>>      start
>>>>      abort: known exception
>>>> +  (?)
>>> Same question as the other place this happens.
>> Same problem, I randomly get the "abort: error: Cannot assign requested
>> address" message.
>>
>>>>      [255]
>>>>
>>>>      $ hg --config "extensions.t=$abspath" --config 'worker.numcpus=2' \
>>>> _______________________________________________
>>>> Mercurial-devel mailing list
>>>> Mercurial-devel@mercurial-scm.org
>>>> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>>
Augie Fackler - April 28, 2017, 4:38 p.m.
On Fri, Apr 28, 2017 at 12:34 PM, lothiraldan <boris.feld@octobus.net> wrote:
> On 28/04/2017 17:43, Augie Fackler wrote:
>>
>> On Fri, Apr 28, 2017 at 11:36 AM, lothiraldan <boris.feld@octobus.net>
>> wrote:
>>>
>>> On 28/04/2017 16:51, Augie Fackler wrote:

[...]

>> but if adding bash to the command line also works for you that'd be my
>> preference, since it works on more systems (this is the docker you get
>> from Ubuntu 14.04)
>
> That works for me! Do you want me to send a new patch?

I'll push what I've got here (which is two patches, excludes the flaky
test failure changes that weirded me out) and we can iterate from here
as needed. Sound good?
Boris Feld - April 28, 2017, 5:23 p.m.
On 28/04/2017 18:38, Augie Fackler wrote:
> On Fri, Apr 28, 2017 at 12:34 PM, lothiraldan <boris.feld@octobus.net> wrote:
>> On 28/04/2017 17:43, Augie Fackler wrote:
>>> On Fri, Apr 28, 2017 at 11:36 AM, lothiraldan <boris.feld@octobus.net>
>>> wrote:
>>>> On 28/04/2017 16:51, Augie Fackler wrote:
> [...]
>
>>> but if adding bash to the command line also works for you that'd be my
>>> preference, since it works on more systems (this is the docker you get
>>> from Ubuntu 14.04)
>> That works for me! Do you want me to send a new patch?
> I'll push what I've got here (which is two patches, excludes the flaky
> test failure changes that weirded me out) and we can iterate from here
> as needed. Sound good?
Yes, sounds good to me. Thanks.

Patch

diff -r 52cca17ac523 -r d49b79b31398 .hgignore
--- a/.hgignore	Mon Apr 24 14:38:30 2017 -0400
+++ b/.hgignore	Tue Apr 25 16:50:01 2017 +0200
@@ -62,6 +62,9 @@ 
 mercurial/osutil.py
 mercurial/parsers.py
 
+# Generated wheels
+wheelhouse/
+
 syntax: regexp
 ^\.pc/
 ^\.(pydev)?project
diff -r 52cca17ac523 -r d49b79b31398 Makefile
--- a/Makefile	Mon Apr 24 14:38:30 2017 -0400
+++ b/Makefile	Tue Apr 25 16:50:01 2017 +0200
@@ -270,6 +270,14 @@ 
 	mkdir -p packages/centos7
 	contrib/dockerrpm centos7
 
+linux-wheels: linux-wheels-x86_64 linux-wheels-i686
+	
+linux-wheels-x86_64:
+	docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src quay.io/pypa/manylinux1_x86_64 /src/contrib/build-linux-wheels.sh
+	
+linux-wheels-i686:
+	docker run -e "HGTEST_JOBS=$(shell nproc)" --rm -ti -v `pwd`:/src quay.io/pypa/manylinux1_i686 linux32 /src/contrib/build-linux-wheels.sh
+
 .PHONY: help all local build doc cleanbutpackages clean install install-bin \
 	install-doc install-home install-home-bin install-home-doc \
 	dist dist-notests check tests check-code update-pot \
@@ -278,4 +286,5 @@ 
 	docker-ubuntu-xenial docker-ubuntu-xenial-ppa \
 	docker-ubuntu-yakkety docker-ubuntu-yakkety-ppa \
 	fedora20 docker-fedora20 fedora21 docker-fedora21 \
-	centos5 docker-centos5 centos6 docker-centos6 centos7 docker-centos7
+	centos5 docker-centos5 centos6 docker-centos6 centos7 docker-centos7 \
+	linux-wheels
diff -r 52cca17ac523 -r d49b79b31398 contrib/build-linux-wheels.sh
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/contrib/build-linux-wheels.sh	Tue Apr 25 16:50:01 2017 +0200
@@ -0,0 +1,34 @@ 
+#!/bin/bash
+# This file is directly inspired by
+# https://github.com/pypa/python-manylinux-demo/blob/master/travis/build-wheels.sh
+set -e -x
+
+PYTHON_TARGETS=$(ls -d /opt/python/cp2*/bin)
+
+# Create an user for the tests
+useradd hgbuilder
+
+# Bypass uid/gid problems
+cp -R /src /io && chown -R hgbuilder:hgbuilder /io
+
+# Compile wheels for Python 2.X
+for PYBIN in $PYTHON_TARGETS; do
+    "${PYBIN}/pip" wheel /io/ -w wheelhouse/
+done
+
+# Bundle external shared libraries into the wheels with
+# auditwheel (https://github.com/pypa/auditwheel) repair.
+# It also fix the ABI tag on the wheel making it pip installable.
+for whl in wheelhouse/*.whl; do
+    auditwheel repair "$whl" -w /src/wheelhouse/
+done
+
+# Install packages and run the tests for all Python versions
+cd /io/tests/
+
+for PYBIN in $PYTHON_TARGETS; do
+    # Install mercurial wheel as root
+    "${PYBIN}/pip" install mercurial --no-index -f /src/wheelhouse
+    # But run tests as hgbuilder user (non-root)
+    su hgbuilder -c "\"${PYBIN}/python\" /io/tests/run-tests.py --with-hg=\"${PYBIN}/hg\" --blacklist=/io/contrib/linux-wheel-centos5-blacklist"
+done
diff -r 52cca17ac523 -r d49b79b31398 contrib/linux-wheel-centos5-blacklist
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/contrib/linux-wheel-centos5-blacklist	Tue Apr 25 16:50:01 2017 +0200
@@ -0,0 +1,3 @@ 
+test-convert-git.t
+test-subrepo-git.t
+test-patchbomb-tls.t
diff -r 52cca17ac523 -r d49b79b31398 tests/test-clonebundles.t
--- a/tests/test-clonebundles.t	Mon Apr 24 14:38:30 2017 -0400
+++ b/tests/test-clonebundles.t	Tue Apr 25 16:50:01 2017 +0200
@@ -61,7 +61,7 @@ 
   $ echo "http://localhost:$HGPORT1/bundle.hg" > server/.hg/clonebundles.manifest
   $ hg clone http://localhost:$HGPORT server-not-runner
   applying clone bundle from http://localhost:$HGPORT1/bundle.hg
-  error fetching bundle: (.* refused.*|Protocol not supported) (re)
+  error fetching bundle: (.* refused.*|Protocol not supported|Cannot assign requested address) (re)
   abort: error applying bundle
   (if this error persists, consider contacting the server operator or disable clone bundles via "--config ui.clonebundles=false")
   [255]
diff -r 52cca17ac523 -r d49b79b31398 tests/test-http-bad-server.t
--- a/tests/test-http-bad-server.t	Mon Apr 24 14:38:30 2017 -0400
+++ b/tests/test-http-bad-server.t	Tue Apr 25 16:50:01 2017 +0200
@@ -36,6 +36,7 @@ 
   $ hg clone http://localhost:$HGPORT/ clone
   abort: error: Connection reset by peer (no-windows !)
   abort: error: An existing connection was forcibly closed by the remote host (windows !)
+  (?)
   [255]
 
 (The server exits on its own, but there is a race between that and starting a new server.
@@ -223,7 +224,7 @@ 
   readline(92 from -1) -> (41) content-type: application/mercurial-0.1\r\n
   readline(51 from -1) -> (19) vary: X-HgProto-1\r\n
   readline(32 from -1) -> (19) x-hgargs-post: 28\r\n
-  readline(13 from -1) -> (13) x-hgproto-1: 
+  readline(13 from -1) -> (13) x-hgproto-1:
   read limit reached; closing socket
   readline\(315 from (-1|65537)\) -> \(27\) POST /\?cmd=batch HTTP/1.1\\r\\n (re)
   readline(288 from -1) -> (27) Accept-Encoding: identity\r\n
diff -r 52cca17ac523 -r d49b79b31398 tests/test-http-proxy.t
--- a/tests/test-http-proxy.t	Mon Apr 24 14:38:30 2017 -0400
+++ b/tests/test-http-proxy.t	Tue Apr 25 16:50:01 2017 +0200
@@ -87,7 +87,7 @@ 
 misconfigured hosts)
 
   $ http_proxy=localhost:$HGPORT2 hg clone --config http_proxy.always=True http://localhost:$HGPORT/ f
-  abort: error: (Connection refused|Protocol not supported|.* actively refused it) (re)
+  abort: error: (Connection refused|Protocol not supported|.* actively refused it|Cannot assign requested address) (re)
   [255]
 
 do not use the proxy if it is in the no list
diff -r 52cca17ac523 -r d49b79b31398 tests/test-worker.t
--- a/tests/test-worker.t	Mon Apr 24 14:38:30 2017 -0400
+++ b/tests/test-worker.t	Tue Apr 25 16:50:01 2017 +0200
@@ -78,6 +78,7 @@ 
   > test 100000.0 abort
   start
   abort: known exception
+  (?)
   [255]
 
   $ hg --config "extensions.t=$abspath" --config 'worker.numcpus=2' \