Patchwork [4,of,4] mpatch: implement cffi version of mpatch

login
register
mail settings
Submitter Maciej Fijalkowski
Date June 8, 2016, 7:59 a.m.
Message ID <037fce72309155ba3806.1465372779@brick.arcode.com>
Download mbox | patch
Permalink /patch/15434/
State Accepted
Headers show

Comments

Maciej Fijalkowski - June 8, 2016, 7:59 a.m.
# HG changeset patch
# User Maciej Fijalkowski <fijall@gmail.com>
# Date 1465372757 -7200
#      Wed Jun 08 09:59:17 2016 +0200
# Node ID 037fce72309155ba3806753d4f1dc3a7db71ee39
# Parent  26a4063b4f1f36607dd673849449292ebc34a91d
mpatch: implement cffi version of mpatch

This is a first go at working, but not 100% efficient version of mpatch
for PyPy that uses cffi. Additional improvements might follow, but it
already makes mercurial far more usable under pypy.
Augie Fackler - June 10, 2016, 3:28 a.m.
On Wed, Jun 08, 2016 at 09:59:39AM +0200, Maciej Fijalkowski wrote:
> # HG changeset patch
> # User Maciej Fijalkowski <fijall@gmail.com>
> # Date 1465372757 -7200
> #      Wed Jun 08 09:59:17 2016 +0200
> # Node ID 037fce72309155ba3806753d4f1dc3a7db71ee39
> # Parent  26a4063b4f1f36607dd673849449292ebc34a91d
> mpatch: implement cffi version of mpatch

I'm happy with these, but I tripped over a nasty rebase and then tests
started failing, so I must have screwed up. Can you take a look at the
failures present in 61bec5d from http://hg.durin42.com/hg-wip/?

http://hg.durin42.com/hg-wip/rev/66c2d32dcba4 in particular took a
rough rebase that I suspect I messed up.

Patch 4 also flunks check-code, see below.

>
> This is a first go at working, but not 100% efficient version of mpatch
> for PyPy that uses cffi. Additional improvements might follow, but it
> already makes mercurial far more usable under pypy.
>
> diff -r 26a4063b4f1f -r 037fce723091 mercurial/build_mpatch_cffi.py
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/mercurial/build_mpatch_cffi.py	Wed Jun 08 09:59:17 2016 +0200

[elided]

> diff -r 26a4063b4f1f -r 037fce723091 mercurial/pure/mpatch.py
> --- a/mercurial/pure/mpatch.py	Tue Jun 07 15:35:58 2016 +0200
> +++ b/mercurial/pure/mpatch.py	Wed Jun 08 09:59:17 2016 +0200
> @@ -9,6 +9,8 @@
>
>  import struct
>
> +from .. import modulepolicy, policynocffi
> +
>  from . import pycompat
>  stringio = pycompat.stringio
>
> @@ -125,3 +127,43 @@
>
>      outlen += orig - last
>      return outlen
> +
> +if modulepolicy not in policynocffi:
> +    try:
> +        from _mpatch_cffi import ffi, lib
> +    except ImportError:
> +        if modulepolicy == 'cffi': # strict cffi import
> +            raise
> +    else:
> +        @ffi.def_extern()
> +        def cffi_get_next_item(arg, pos):

could we call this something like cffigetnext() to make check-code happy?

(I like this name as-is, but the linter disagrees with us.)
Maciej Fijalkowski - June 15, 2016, 9:32 a.m.
Hi Augie.

The problem seems to be _demandimport being either buggy or me not
using it correctly. What's the way these days to import something from
mercurial in pure submodule?

On Fri, Jun 10, 2016 at 5:28 AM, Augie Fackler <raf@durin42.com> wrote:
> On Wed, Jun 08, 2016 at 09:59:39AM +0200, Maciej Fijalkowski wrote:
>> # HG changeset patch
>> # User Maciej Fijalkowski <fijall@gmail.com>
>> # Date 1465372757 -7200
>> #      Wed Jun 08 09:59:17 2016 +0200
>> # Node ID 037fce72309155ba3806753d4f1dc3a7db71ee39
>> # Parent  26a4063b4f1f36607dd673849449292ebc34a91d
>> mpatch: implement cffi version of mpatch
>
> I'm happy with these, but I tripped over a nasty rebase and then tests
> started failing, so I must have screwed up. Can you take a look at the
> failures present in 61bec5d from http://hg.durin42.com/hg-wip/?
>
> http://hg.durin42.com/hg-wip/rev/66c2d32dcba4 in particular took a
> rough rebase that I suspect I messed up.
>
> Patch 4 also flunks check-code, see below.
>
>>
>> This is a first go at working, but not 100% efficient version of mpatch
>> for PyPy that uses cffi. Additional improvements might follow, but it
>> already makes mercurial far more usable under pypy.
>>
>> diff -r 26a4063b4f1f -r 037fce723091 mercurial/build_mpatch_cffi.py
>> --- /dev/null Thu Jan 01 00:00:00 1970 +0000
>> +++ b/mercurial/build_mpatch_cffi.py  Wed Jun 08 09:59:17 2016 +0200
>
> [elided]
>
>> diff -r 26a4063b4f1f -r 037fce723091 mercurial/pure/mpatch.py
>> --- a/mercurial/pure/mpatch.py        Tue Jun 07 15:35:58 2016 +0200
>> +++ b/mercurial/pure/mpatch.py        Wed Jun 08 09:59:17 2016 +0200
>> @@ -9,6 +9,8 @@
>>
>>  import struct
>>
>> +from .. import modulepolicy, policynocffi
>> +
>>  from . import pycompat
>>  stringio = pycompat.stringio
>>
>> @@ -125,3 +127,43 @@
>>
>>      outlen += orig - last
>>      return outlen
>> +
>> +if modulepolicy not in policynocffi:
>> +    try:
>> +        from _mpatch_cffi import ffi, lib
>> +    except ImportError:
>> +        if modulepolicy == 'cffi': # strict cffi import
>> +            raise
>> +    else:
>> +        @ffi.def_extern()
>> +        def cffi_get_next_item(arg, pos):
>
> could we call this something like cffigetnext() to make check-code happy?
>
> (I like this name as-is, but the linter disagrees with us.)
Augie Fackler - June 15, 2016, 12:25 p.m.
(+some people that understand our recent importing changes, hopefully one of them can help you)

> On Jun 15, 2016, at 5:32 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
> 
> Hi Augie.
> 
> The problem seems to be _demandimport being either buggy or me not
> using it correctly. What's the way these days to import something from
> mercurial in pure submodule?
> 
> On Fri, Jun 10, 2016 at 5:28 AM, Augie Fackler <raf@durin42.com> wrote:
>> On Wed, Jun 08, 2016 at 09:59:39AM +0200, Maciej Fijalkowski wrote:
>>> # HG changeset patch
>>> # User Maciej Fijalkowski <fijall@gmail.com>
>>> # Date 1465372757 -7200
>>> #      Wed Jun 08 09:59:17 2016 +0200
>>> # Node ID 037fce72309155ba3806753d4f1dc3a7db71ee39
>>> # Parent  26a4063b4f1f36607dd673849449292ebc34a91d
>>> mpatch: implement cffi version of mpatch
>> 
>> I'm happy with these, but I tripped over a nasty rebase and then tests
>> started failing, so I must have screwed up. Can you take a look at the
>> failures present in 61bec5d from http://hg.durin42.com/hg-wip/?
>> 
>> http://hg.durin42.com/hg-wip/rev/66c2d32dcba4 in particular took a
>> rough rebase that I suspect I messed up.
>> 
>> Patch 4 also flunks check-code, see below.
>> 
>>> 
>>> This is a first go at working, but not 100% efficient version of mpatch
>>> for PyPy that uses cffi. Additional improvements might follow, but it
>>> already makes mercurial far more usable under pypy.
>>> 
>>> diff -r 26a4063b4f1f -r 037fce723091 mercurial/build_mpatch_cffi.py
>>> --- /dev/null Thu Jan 01 00:00:00 1970 +0000
>>> +++ b/mercurial/build_mpatch_cffi.py  Wed Jun 08 09:59:17 2016 +0200
>> 
>> [elided]
>> 
>>> diff -r 26a4063b4f1f -r 037fce723091 mercurial/pure/mpatch.py
>>> --- a/mercurial/pure/mpatch.py        Tue Jun 07 15:35:58 2016 +0200
>>> +++ b/mercurial/pure/mpatch.py        Wed Jun 08 09:59:17 2016 +0200
>>> @@ -9,6 +9,8 @@
>>> 
>>> import struct
>>> 
>>> +from .. import modulepolicy, policynocffi
>>> +
>>> from . import pycompat
>>> stringio = pycompat.stringio
>>> 
>>> @@ -125,3 +127,43 @@
>>> 
>>>     outlen += orig - last
>>>     return outlen
>>> +
>>> +if modulepolicy not in policynocffi:
>>> +    try:
>>> +        from _mpatch_cffi import ffi, lib
>>> +    except ImportError:
>>> +        if modulepolicy == 'cffi': # strict cffi import
>>> +            raise
>>> +    else:
>>> +        @ffi.def_extern()
>>> +        def cffi_get_next_item(arg, pos):
>> 
>> could we call this something like cffigetnext() to make check-code happy?
>> 
>> (I like this name as-is, but the linter disagrees with us.)
timeless - June 17, 2016, 1:19 a.m.
I've hit something like this, I think I've tried a couple of things, I'm
not sure I found anything satisfactory. I think I was considering a change
because of edges like this...
On Jun 15, 2016 5:25 AM, "Augie Fackler" <raf@durin42.com> wrote:

> (+some people that understand our recent importing changes, hopefully one
> of them can help you)
>
> > On Jun 15, 2016, at 5:32 AM, Maciej Fijalkowski <fijall@gmail.com>
> wrote:
> >
> > Hi Augie.
> >
> > The problem seems to be _demandimport being either buggy or me not
> > using it correctly. What's the way these days to import something from
> > mercurial in pure submodule?
> >
> > On Fri, Jun 10, 2016 at 5:28 AM, Augie Fackler <raf@durin42.com> wrote:
> >> On Wed, Jun 08, 2016 at 09:59:39AM +0200, Maciej Fijalkowski wrote:
> >>> # HG changeset patch
> >>> # User Maciej Fijalkowski <fijall@gmail.com>
> >>> # Date 1465372757 -7200
> >>> #      Wed Jun 08 09:59:17 2016 +0200
> >>> # Node ID 037fce72309155ba3806753d4f1dc3a7db71ee39
> >>> # Parent  26a4063b4f1f36607dd673849449292ebc34a91d
> >>> mpatch: implement cffi version of mpatch
> >>
> >> I'm happy with these, but I tripped over a nasty rebase and then tests
> >> started failing, so I must have screwed up. Can you take a look at the
> >> failures present in 61bec5d from http://hg.durin42.com/hg-wip/?
> >>
> >> http://hg.durin42.com/hg-wip/rev/66c2d32dcba4 in particular took a
> >> rough rebase that I suspect I messed up.
> >>
> >> Patch 4 also flunks check-code, see below.
> >>
> >>>
> >>> This is a first go at working, but not 100% efficient version of mpatch
> >>> for PyPy that uses cffi. Additional improvements might follow, but it
> >>> already makes mercurial far more usable under pypy.
> >>>
> >>> diff -r 26a4063b4f1f -r 037fce723091 mercurial/build_mpatch_cffi.py
> >>> --- /dev/null Thu Jan 01 00:00:00 1970 +0000
> >>> +++ b/mercurial/build_mpatch_cffi.py  Wed Jun 08 09:59:17 2016 +0200
> >>
> >> [elided]
> >>
> >>> diff -r 26a4063b4f1f -r 037fce723091 mercurial/pure/mpatch.py
> >>> --- a/mercurial/pure/mpatch.py        Tue Jun 07 15:35:58 2016 +0200
> >>> +++ b/mercurial/pure/mpatch.py        Wed Jun 08 09:59:17 2016 +0200
> >>> @@ -9,6 +9,8 @@
> >>>
> >>> import struct
> >>>
> >>> +from .. import modulepolicy, policynocffi
> >>> +
> >>> from . import pycompat
> >>> stringio = pycompat.stringio
> >>>
> >>> @@ -125,3 +127,43 @@
> >>>
> >>>     outlen += orig - last
> >>>     return outlen
> >>> +
> >>> +if modulepolicy not in policynocffi:
> >>> +    try:
> >>> +        from _mpatch_cffi import ffi, lib
> >>> +    except ImportError:
> >>> +        if modulepolicy == 'cffi': # strict cffi import
> >>> +            raise
> >>> +    else:
> >>> +        @ffi.def_extern()
> >>> +        def cffi_get_next_item(arg, pos):
> >>
> >> could we call this something like cffigetnext() to make check-code
> happy?
> >>
> >> (I like this name as-is, but the linter disagrees with us.)
>
>
Maciej Fijalkowski - June 20, 2016, 3:04 p.m.
er ok, so there is no good way to do imports that way? At least I
didn't find any that would not make the linter barf (using from
mercurial). Can we do something that special cases ".." here?

On Fri, Jun 17, 2016 at 3:19 AM, timeless <timeless@gmail.com> wrote:
> I've hit something like this, I think I've tried a couple of things, I'm not
> sure I found anything satisfactory. I think I was considering a change
> because of edges like this...
>
> On Jun 15, 2016 5:25 AM, "Augie Fackler" <raf@durin42.com> wrote:
>>
>> (+some people that understand our recent importing changes, hopefully one
>> of them can help you)
>>
>> > On Jun 15, 2016, at 5:32 AM, Maciej Fijalkowski <fijall@gmail.com>
>> > wrote:
>> >
>> > Hi Augie.
>> >
>> > The problem seems to be _demandimport being either buggy or me not
>> > using it correctly. What's the way these days to import something from
>> > mercurial in pure submodule?
>> >
>> > On Fri, Jun 10, 2016 at 5:28 AM, Augie Fackler <raf@durin42.com> wrote:
>> >> On Wed, Jun 08, 2016 at 09:59:39AM +0200, Maciej Fijalkowski wrote:
>> >>> # HG changeset patch
>> >>> # User Maciej Fijalkowski <fijall@gmail.com>
>> >>> # Date 1465372757 -7200
>> >>> #      Wed Jun 08 09:59:17 2016 +0200
>> >>> # Node ID 037fce72309155ba3806753d4f1dc3a7db71ee39
>> >>> # Parent  26a4063b4f1f36607dd673849449292ebc34a91d
>> >>> mpatch: implement cffi version of mpatch
>> >>
>> >> I'm happy with these, but I tripped over a nasty rebase and then tests
>> >> started failing, so I must have screwed up. Can you take a look at the
>> >> failures present in 61bec5d from http://hg.durin42.com/hg-wip/?
>> >>
>> >> http://hg.durin42.com/hg-wip/rev/66c2d32dcba4 in particular took a
>> >> rough rebase that I suspect I messed up.
>> >>
>> >> Patch 4 also flunks check-code, see below.
>> >>
>> >>>
>> >>> This is a first go at working, but not 100% efficient version of
>> >>> mpatch
>> >>> for PyPy that uses cffi. Additional improvements might follow, but it
>> >>> already makes mercurial far more usable under pypy.
>> >>>
>> >>> diff -r 26a4063b4f1f -r 037fce723091 mercurial/build_mpatch_cffi.py
>> >>> --- /dev/null Thu Jan 01 00:00:00 1970 +0000
>> >>> +++ b/mercurial/build_mpatch_cffi.py  Wed Jun 08 09:59:17 2016 +0200
>> >>
>> >> [elided]
>> >>
>> >>> diff -r 26a4063b4f1f -r 037fce723091 mercurial/pure/mpatch.py
>> >>> --- a/mercurial/pure/mpatch.py        Tue Jun 07 15:35:58 2016 +0200
>> >>> +++ b/mercurial/pure/mpatch.py        Wed Jun 08 09:59:17 2016 +0200
>> >>> @@ -9,6 +9,8 @@
>> >>>
>> >>> import struct
>> >>>
>> >>> +from .. import modulepolicy, policynocffi
>> >>> +
>> >>> from . import pycompat
>> >>> stringio = pycompat.stringio
>> >>>
>> >>> @@ -125,3 +127,43 @@
>> >>>
>> >>>     outlen += orig - last
>> >>>     return outlen
>> >>> +
>> >>> +if modulepolicy not in policynocffi:
>> >>> +    try:
>> >>> +        from _mpatch_cffi import ffi, lib
>> >>> +    except ImportError:
>> >>> +        if modulepolicy == 'cffi': # strict cffi import
>> >>> +            raise
>> >>> +    else:
>> >>> +        @ffi.def_extern()
>> >>> +        def cffi_get_next_item(arg, pos):
>> >>
>> >> could we call this something like cffigetnext() to make check-code
>> >> happy?
>> >>
>> >> (I like this name as-is, but the linter disagrees with us.)
>>
>
Yuya Nishihara - June 21, 2016, 11:55 a.m.
On Mon, 20 Jun 2016 17:04:25 +0200, Maciej Fijalkowski wrote:
> er ok, so there is no good way to do imports that way? At least I
> didn't find any that would not make the linter barf (using from
> mercurial). Can we do something that special cases ".." here?

> >> >>> +from .. import modulepolicy, policynocffi
> >> >>> +
> >> >>> from . import pycompat

from . import (
    policy,
    pycompat,
)

I don't know if it works for mercurial/__init__.py, but we won't have to
care because modulepolicy constants has been moved to policy.py.
Maciej Fijalkowski - June 22, 2016, 7:06 a.m.
This seems to be the diff necessary (to Augie's wip branch)

https://paste.pound-python.org/show/CMz1afIMKS0r7369G3Jr/

This fails test-check-py3 for stupid reasons.

Augie: how should I submit that patch, given all the rebasing and stuff?

Best regards,
Maciej Fijalkowski

On Tue, Jun 21, 2016 at 1:55 PM, Yuya Nishihara <yuya@tcha.org> wrote:
> On Mon, 20 Jun 2016 17:04:25 +0200, Maciej Fijalkowski wrote:
>> er ok, so there is no good way to do imports that way? At least I
>> didn't find any that would not make the linter barf (using from
>> mercurial). Can we do something that special cases ".." here?
>
>> >> >>> +from .. import modulepolicy, policynocffi
>> >> >>> +
>> >> >>> from . import pycompat
>
> from . import (
>     policy,
>     pycompat,
> )
>
> I don't know if it works for mercurial/__init__.py, but we won't have to
> care because modulepolicy constants has been moved to policy.py.
Martijn Pieters - June 22, 2016, 12:24 p.m.
On 22 June 2016 at 08:06, Maciej Fijalkowski <fijall@gmail.com> wrote:
> This seems to be the diff necessary (to Augie's wip branch)
>
> https://paste.pound-python.org/show/CMz1afIMKS0r7369G3Jr/
>
> This fails test-check-py3 for stupid reasons.

Might it be because

  from __future__ import absolute_import

is missing from the top of the module?

> Augie: how should I submit that patch, given all the rebasing and stuff?
>
> Best regards,
> Maciej Fijalkowski
>
> On Tue, Jun 21, 2016 at 1:55 PM, Yuya Nishihara <yuya@tcha.org> wrote:
>> On Mon, 20 Jun 2016 17:04:25 +0200, Maciej Fijalkowski wrote:
>>> er ok, so there is no good way to do imports that way? At least I
>>> didn't find any that would not make the linter barf (using from
>>> mercurial). Can we do something that special cases ".." here?
>>
>>> >> >>> +from .. import modulepolicy, policynocffi
>>> >> >>> +
>>> >> >>> from . import pycompat
>>
>> from . import (
>>     policy,
>>     pycompat,
>> )
>>
>> I don't know if it works for mercurial/__init__.py, but we won't have to
>> care because modulepolicy constants has been moved to policy.py.
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Augie Fackler - June 22, 2016, 3:11 p.m.
> On Jun 22, 2016, at 3:06 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
> 
> This seems to be the diff necessary (to Augie's wip branch)
> 
> https://paste.pound-python.org/show/CMz1afIMKS0r7369G3Jr/
> 
> This fails test-check-py3 for stupid reasons.
> 
> Augie: how should I submit that patch, given all the rebasing and stuff?

Apply the patch, use histedit to fold it into the old version of 4 of 4, rebase to @, and mail again.

(That’s my advice, but I’m only half paying attention this week.)

> 
> Best regards,
> Maciej Fijalkowski
> 
> On Tue, Jun 21, 2016 at 1:55 PM, Yuya Nishihara <yuya@tcha.org> wrote:
>> On Mon, 20 Jun 2016 17:04:25 +0200, Maciej Fijalkowski wrote:
>>> er ok, so there is no good way to do imports that way? At least I
>>> didn't find any that would not make the linter barf (using from
>>> mercurial). Can we do something that special cases ".." here?
>> 
>>>>>>>> +from .. import modulepolicy, policynocffi
>>>>>>>> +
>>>>>>>> from . import pycompat
>> 
>> from . import (
>>    policy,
>>    pycompat,
>> )
>> 
>> I don't know if it works for mercurial/__init__.py, but we won't have to
>> care because modulepolicy constants has been moved to policy.py.
Maciej Fijalkowski - June 23, 2016, 6:51 a.m.
ok, the next problem is:

import os
import cffi

"Imports not lexically sorted"

import cffi
import os

"Stdlib import follows non-stdlib import"

what do I do?

On Wed, Jun 22, 2016 at 5:11 PM, Augie Fackler <raf@durin42.com> wrote:
>
>> On Jun 22, 2016, at 3:06 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
>>
>> This seems to be the diff necessary (to Augie's wip branch)
>>
>> https://paste.pound-python.org/show/CMz1afIMKS0r7369G3Jr/
>>
>> This fails test-check-py3 for stupid reasons.
>>
>> Augie: how should I submit that patch, given all the rebasing and stuff?
>
> Apply the patch, use histedit to fold it into the old version of 4 of 4, rebase to @, and mail again.
>
> (That’s my advice, but I’m only half paying attention this week.)
>
>>
>> Best regards,
>> Maciej Fijalkowski
>>
>> On Tue, Jun 21, 2016 at 1:55 PM, Yuya Nishihara <yuya@tcha.org> wrote:
>>> On Mon, 20 Jun 2016 17:04:25 +0200, Maciej Fijalkowski wrote:
>>>> er ok, so there is no good way to do imports that way? At least I
>>>> didn't find any that would not make the linter barf (using from
>>>> mercurial). Can we do something that special cases ".." here?
>>>
>>>>>>>>> +from .. import modulepolicy, policynocffi
>>>>>>>>> +
>>>>>>>>> from . import pycompat
>>>
>>> from . import (
>>>    policy,
>>>    pycompat,
>>> )
>>>
>>> I don't know if it works for mercurial/__init__.py, but we won't have to
>>> care because modulepolicy constants has been moved to policy.py.
>
Augie Fackler - June 23, 2016, 1:23 p.m.
> On Jun 23, 2016, at 2:51 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
> 
> ok, the next problem is:
> 
> import os
> import cffi
> 
> "Imports not lexically sorted"
> 
> import cffi
> import os
> 
> "Stdlib import follows non-stdlib import"
> 
> what do I do?

Smells like a bug in the import checker, where it’s not recognizing cffi as part of the sodlib. I’ll add cffi to the whitelist.

AF

> 
> On Wed, Jun 22, 2016 at 5:11 PM, Augie Fackler <raf@durin42.com> wrote:
>> 
>>> On Jun 22, 2016, at 3:06 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
>>> 
>>> This seems to be the diff necessary (to Augie's wip branch)
>>> 
>>> https://paste.pound-python.org/show/CMz1afIMKS0r7369G3Jr/
>>> 
>>> This fails test-check-py3 for stupid reasons.
>>> 
>>> Augie: how should I submit that patch, given all the rebasing and stuff?
>> 
>> Apply the patch, use histedit to fold it into the old version of 4 of 4, rebase to @, and mail again.
>> 
>> (That’s my advice, but I’m only half paying attention this week.)
>> 
>>> 
>>> Best regards,
>>> Maciej Fijalkowski
>>> 
>>> On Tue, Jun 21, 2016 at 1:55 PM, Yuya Nishihara <yuya@tcha.org> wrote:
>>>> On Mon, 20 Jun 2016 17:04:25 +0200, Maciej Fijalkowski wrote:
>>>>> er ok, so there is no good way to do imports that way? At least I
>>>>> didn't find any that would not make the linter barf (using from
>>>>> mercurial). Can we do something that special cases ".." here?
>>>> 
>>>>>>>>>> +from .. import modulepolicy, policynocffi
>>>>>>>>>> +
>>>>>>>>>> from . import pycompat
>>>> 
>>>> from . import (
>>>>   policy,
>>>>   pycompat,
>>>> )
>>>> 
>>>> I don't know if it works for mercurial/__init__.py, but we won't have to
>>>> care because modulepolicy constants has been moved to policy.py.
>>
Maciej Fijalkowski - June 27, 2016, 8:53 a.m.
On Thu, Jun 23, 2016 at 3:23 PM, Augie Fackler <raf@durin42.com> wrote:
>
>> On Jun 23, 2016, at 2:51 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
>>
>> ok, the next problem is:
>>
>> import os
>> import cffi
>>
>> "Imports not lexically sorted"
>>
>> import cffi
>> import os
>>
>> "Stdlib import follows non-stdlib import"
>>
>> what do I do?
>
> Smells like a bug in the import checker, where it’s not recognizing cffi as part of the sodlib. I’ll add cffi to the whitelist.
>
> AF

Thanks. Is there anything else or is the patch good to go?

>
>>
>> On Wed, Jun 22, 2016 at 5:11 PM, Augie Fackler <raf@durin42.com> wrote:
>>>
>>>> On Jun 22, 2016, at 3:06 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
>>>>
>>>> This seems to be the diff necessary (to Augie's wip branch)
>>>>
>>>> https://paste.pound-python.org/show/CMz1afIMKS0r7369G3Jr/
>>>>
>>>> This fails test-check-py3 for stupid reasons.
>>>>
>>>> Augie: how should I submit that patch, given all the rebasing and stuff?
>>>
>>> Apply the patch, use histedit to fold it into the old version of 4 of 4, rebase to @, and mail again.
>>>
>>> (That’s my advice, but I’m only half paying attention this week.)
>>>
>>>>
>>>> Best regards,
>>>> Maciej Fijalkowski
>>>>
>>>> On Tue, Jun 21, 2016 at 1:55 PM, Yuya Nishihara <yuya@tcha.org> wrote:
>>>>> On Mon, 20 Jun 2016 17:04:25 +0200, Maciej Fijalkowski wrote:
>>>>>> er ok, so there is no good way to do imports that way? At least I
>>>>>> didn't find any that would not make the linter barf (using from
>>>>>> mercurial). Can we do something that special cases ".." here?
>>>>>
>>>>>>>>>>> +from .. import modulepolicy, policynocffi
>>>>>>>>>>> +
>>>>>>>>>>> from . import pycompat
>>>>>
>>>>> from . import (
>>>>>   policy,
>>>>>   pycompat,
>>>>> )
>>>>>
>>>>> I don't know if it works for mercurial/__init__.py, but we won't have to
>>>>> care because modulepolicy constants has been moved to policy.py.
>>>
>
Augie Fackler - June 27, 2016, 2:41 p.m.
> On Jun 27, 2016, at 4:53 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
> 
> On Thu, Jun 23, 2016 at 3:23 PM, Augie Fackler <raf@durin42.com> wrote:
>> 
>>> On Jun 23, 2016, at 2:51 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
>>> 
>>> ok, the next problem is:
>>> 
>>> import os
>>> import cffi
>>> 
>>> "Imports not lexically sorted"
>>> 
>>> import cffi
>>> import os
>>> 
>>> "Stdlib import follows non-stdlib import"
>>> 
>>> what do I do?
>> 
>> Smells like a bug in the import checker, where it’s not recognizing cffi as part of the sodlib. I’ll add cffi to the whitelist.
>> 
>> AF
> 
> Thanks. Is there anything else or is the patch good to go?

I think it’s good to go. Where is the current version of this patch? (I think it’s changed since this version went to the list.)

Sorry for all the nightmares you’re tripping over in this series - this has been pretty rough as a way to get started contributing it Mercurial. :(

> 
>> 
>>> 
>>> On Wed, Jun 22, 2016 at 5:11 PM, Augie Fackler <raf@durin42.com> wrote:
>>>> 
>>>>> On Jun 22, 2016, at 3:06 AM, Maciej Fijalkowski <fijall@gmail.com> wrote:
>>>>> 
>>>>> This seems to be the diff necessary (to Augie's wip branch)
>>>>> 
>>>>> https://paste.pound-python.org/show/CMz1afIMKS0r7369G3Jr/
>>>>> 
>>>>> This fails test-check-py3 for stupid reasons.
>>>>> 
>>>>> Augie: how should I submit that patch, given all the rebasing and stuff?
>>>> 
>>>> Apply the patch, use histedit to fold it into the old version of 4 of 4, rebase to @, and mail again.
>>>> 
>>>> (That’s my advice, but I’m only half paying attention this week.)
>>>> 
>>>>> 
>>>>> Best regards,
>>>>> Maciej Fijalkowski
>>>>> 
>>>>> On Tue, Jun 21, 2016 at 1:55 PM, Yuya Nishihara <yuya@tcha.org> wrote:
>>>>>> On Mon, 20 Jun 2016 17:04:25 +0200, Maciej Fijalkowski wrote:
>>>>>>> er ok, so there is no good way to do imports that way? At least I
>>>>>>> didn't find any that would not make the linter barf (using from
>>>>>>> mercurial). Can we do something that special cases ".." here?
>>>>>> 
>>>>>>>>>>>> +from .. import modulepolicy, policynocffi
>>>>>>>>>>>> +
>>>>>>>>>>>> from . import pycompat
>>>>>> 
>>>>>> from . import (
>>>>>>  policy,
>>>>>>  pycompat,
>>>>>> )
>>>>>> 
>>>>>> I don't know if it works for mercurial/__init__.py, but we won't have to
>>>>>> care because modulepolicy constants has been moved to policy.py.
>>>> 
>>

Patch

diff -r 26a4063b4f1f -r 037fce723091 mercurial/build_mpatch_cffi.py
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/mercurial/build_mpatch_cffi.py	Wed Jun 08 09:59:17 2016 +0200
@@ -0,0 +1,31 @@ 
+import os
+from cffi import FFI
+
+ffi = FFI()
+mpatch_c = os.path.join(os.path.dirname(__file__), 'mpatch.c')
+ffi.set_source("_mpatch_cffi", open(mpatch_c).read(),
+               include_dirs=["mercurial"])
+ffi.cdef("""
+
+struct frag {
+       int start, end, len;
+       const char *data;
+};
+
+struct flist {
+       struct frag *base, *head, *tail;
+};
+
+extern "Python" struct flist* cffi_get_next_item(void*, ssize_t);
+
+int decode(const char *bin, size_t len, struct flist** res);
+struct flist *combine(struct flist *a, struct flist *b);
+ssize_t calcsize(size_t len, struct flist *l);
+void lfree(struct flist *a);
+static int apply(char *buf, const char *orig, size_t len, struct flist *l);
+struct flist *fold(void *bins, struct flist* (*get_next_item)(void*, ssize_t),
+                   ssize_t start, ssize_t end);
+""")
+
+if __name__ == '__main__':
+    ffi.compile()
diff -r 26a4063b4f1f -r 037fce723091 mercurial/mpatch.h
--- a/mercurial/mpatch.h	Tue Jun 07 15:35:58 2016 +0200
+++ b/mercurial/mpatch.h	Wed Jun 08 09:59:17 2016 +0200
@@ -8,21 +8,21 @@ 
 #define MPATCH_ERR_INVALID_PATCH -3
 
 struct frag {
-       int start, end, len;
-       const char *data;
+	int start, end, len;
+	const char *data;
 };
 
 struct flist {
-       struct frag *base, *head, *tail;
+	struct frag *base, *head, *tail;
 };
 
-int decode(const char *bin, ssize_t len, struct flist** res);
+int decode(const char *bin, ssize_t len, struct flist **res);
 struct flist *combine(struct flist *a, struct flist *b);
 ssize_t calcsize(ssize_t len, struct flist *l);
 void lfree(struct flist *a);
 int apply(char *buf, const char *orig, ssize_t len, struct flist *l);
-ssize_t _patchedsize(long orig, char* bin, ssize_t patchlen);
+ssize_t _patchedsize(long orig, char *bin, ssize_t patchlen);
 struct flist *fold(void *bins, struct flist* (*get_next_item)(void*, ssize_t),
-	               ssize_t start, ssize_t end);
+					ssize_t start, ssize_t end);
 
-#endif // _HG_MPATCH_H_
\ No newline at end of file
+#endif
diff -r 26a4063b4f1f -r 037fce723091 mercurial/mpatch_module.c
--- a/mercurial/mpatch_module.c	Tue Jun 07 15:35:58 2016 +0200
+++ b/mercurial/mpatch_module.c	Wed Jun 08 09:59:17 2016 +0200
@@ -10,7 +10,7 @@ 
 
 static PyObject *mpatch_Error;
 
-struct flist* cpy_get_next_item(void* bins, ssize_t pos)
+struct flist *cpy_get_item(void *bins, ssize_t pos)
 {
 	const char *buffer;
 	struct flist *res;
@@ -52,7 +52,7 @@ 
 	if (PyObject_AsCharBuffer(text, &in, &inlen))
 		return NULL;
 
-	patch = fold(bins, cpy_get_next_item, 0, len);
+	patch = fold(bins, cpy_get_item, 0, len);
 	if (!patch) {
 		return NULL;
 	}
diff -r 26a4063b4f1f -r 037fce723091 mercurial/pure/mpatch.py
--- a/mercurial/pure/mpatch.py	Tue Jun 07 15:35:58 2016 +0200
+++ b/mercurial/pure/mpatch.py	Wed Jun 08 09:59:17 2016 +0200
@@ -9,6 +9,8 @@ 
 
 import struct
 
+from .. import modulepolicy, policynocffi
+
 from . import pycompat
 stringio = pycompat.stringio
 
@@ -125,3 +127,43 @@ 
 
     outlen += orig - last
     return outlen
+
+if modulepolicy not in policynocffi:
+    try:
+        from _mpatch_cffi import ffi, lib
+    except ImportError:
+        if modulepolicy == 'cffi': # strict cffi import
+            raise
+    else:
+        @ffi.def_extern()
+        def cffi_get_next_item(arg, pos):
+            all, bins = ffi.from_handle(arg)
+            container = ffi.new("struct flist*[1]")
+            to_pass = ffi.new("char[]", str(bins[pos]))
+            all.append(to_pass)
+            r = lib.decode(to_pass, len(to_pass) - 1, container)
+            if r < 0:
+                return ffi.NULL
+            return container[0]
+
+        def patches(text, bins):
+            lgt = len(bins)
+            all = []
+            if not lgt:
+                return text
+            arg = (all, bins)
+            patch = lib.fold(ffi.new_handle(arg), lib.cffi_get_next_item, 0,
+                             lgt)
+            if not patch:
+                raise mpatchError("cannot decode chunk")
+            outlen = lib.calcsize(len(text), patch)
+            if outlen < 0:
+                lib.lfree(patch)
+                raise mpatchError("inconsistency detected")
+            buf = ffi.new("char[]", outlen)
+            if lib.apply(buf, text, len(text), patch) < 0:
+                lib.lfree(patch)
+                raise mpatchError("error applying patches")
+            res = ffi.buffer(buf, outlen)[:]
+            lib.lfree(patch)
+            return res
diff -r 26a4063b4f1f -r 037fce723091 setup.py
--- a/setup.py	Tue Jun 07 15:35:58 2016 +0200
+++ b/setup.py	Wed Jun 08 09:59:17 2016 +0200
@@ -262,7 +262,8 @@ 
 
 
 class hgdist(Distribution):
-    pure = ispypy
+    pure = False
+    cffi = ispypy
 
     global_options = Distribution.global_options + \
                      [('pure', None, "use pure (slow) Python "
@@ -316,6 +317,10 @@ 
 
         if self.distribution.pure:
             self.distribution.ext_modules = []
+        elif self.distribution.cffi:
+            from mercurial import build_mpatch_cffi
+            exts = [build_mpatch_cffi.ffi.distutils_extension()]
+            self.distribution.ext_modules = exts
         else:
             h = os.path.join(get_python_inc(), 'Python.h')
             if not os.path.exists(h):