Patchwork [6,of,6] import-checker: check docstrings in python

login
register
mail settings
Submitter timeless@mozdev.org
Date April 12, 2016, 9:54 p.m.
Message ID <45700c5521d9d77ccaf6.1460498044@waste.org>
Download mbox | patch
Permalink /patch/14566/
State Superseded
Headers show

Comments

timeless@mozdev.org - April 12, 2016, 9:54 p.m.
# HG changeset patch
# User timeless <timeless@mozdev.org>
# Date 1460497442 0
#      Tue Apr 12 21:44:02 2016 +0000
# Node ID 45700c5521d9d77ccaf6fa535ca2f195a52f0b75
# Parent  5ff1ae440687032163d88924da1b5c55394fb581
import-checker: check docstrings in python
Simon Farnsworth - April 13, 2016, 11:37 a.m.
Apart from the nitpicks in other mails, the series looks OK to me.

Out of interest, how many more import problems does this show in the 
current Mercurial codebase?

On 12/04/2016 22:54, timeless wrote:
> # HG changeset patch
> # User timeless <timeless@mozdev.org>
> # Date 1460497442 0
> #      Tue Apr 12 21:44:02 2016 +0000
> # Node ID 45700c5521d9d77ccaf6fa535ca2f195a52f0b75
> # Parent  5ff1ae440687032163d88924da1b5c55394fb581
> import-checker: check docstrings in python
>
> diff --git a/contrib/import-checker.py b/contrib/import-checker.py
> --- a/contrib/import-checker.py
> +++ b/contrib/import-checker.py
> @@ -598,19 +598,41 @@
>       'from __future__ import print_function\\n'
>       example[11] doctest.py 11
>       'from __future__ import print_function\\\\\\n, absolute_import\\n'
> +
> +    >>> lines = [
> +    ... 'def x():',
> +    ... "    ''' # docstring",
> +    ... '    x()',
> +    ... '    >>> x() + 1',
> +    ... '    3',
> +    ... "    '''",
> +    ... '    return 2',
> +    ... ]
> +
> +    >>> test("example.py", lines)
> +    example[4] doctest.py 4
> +    'x() + 1\\n'
>       """
>       inlinepython = 0
>       shpython = 0
>       script = []
> -    prefix = 6
>       t = ''
> +    pystr = ''
>       n = 0
>       linecont = 0
> +    if f.endswith('.t'):
> +        pyinlinestart = b'  >>> '
> +        pyinlinecont = b'  ... '
> +        prefix = 6
> +    else:
> +        pyinlinestart = b'>>> '
> +        pyinlinecont = b'... '
> +        prefix = 4
>       for l in src:
>           n += 1
>           if not l.endswith(b'\n'):
>               l += b'\n'
> -        if l.startswith(b'  >>> '): # python inlines
> +        if l.startswith(pyinlinestart): # python inlines
>               if shpython:
>                   print("Parse Error %s:%d\n" % (f, n))
>               if not inlinepython:
> @@ -620,7 +642,7 @@
>               script.append(l[prefix:])
>               linecont = l.strip('\n').endswith('\\')
>               continue
> -        if l.startswith(b'  ... '): # python inlines
> +        if l.startswith(pyinlinecont): # python inlines
>               script.append(l[prefix:])
>               linecont = l.strip('\n').endswith('\\')
>               continue
> @@ -652,6 +674,36 @@
>               inlinepython = 0
>               linecont = False
>               continue
> +        marker = l.lstrip()[:3]
> +        if pystr:
> +            if marker == pystr:
> +                pystr = ''
> +                if not inlinepython:
> +                    continue
> +                yield ''.join(script), ("%s[%d]" %
> +                       (modname, inlinepython)), t, inlinepython
> +                script = []
> +                inlinepython = 0
> +                linecont = False
> +                continue
> +            pydoc = re.search('^(\s*)>>> ', l)
> +            if pydoc:
> +                pyinlinestart = pydoc.group()
> +                pyinlinecont = pyinlinestart.replace('>>>', '...')
> +                prefix = len(pyinlinestart)
> +                inlinepython = n
> +                t = 'doctest.py'
> +                script.append(l[prefix:])
> +                linecont = l.strip('\n').endswith('\\')
> +                continue
> +            if pystr == l.strip()[-3:]:
> +                pystr = ''
> +                continue
> +        if marker in ("'''", '"""'):
> +            if l.lstrip()[3:].strip().endswith(marker):
> +                continue
> +            pystr = marker
> +            continue
>           if linecont:
>               script.append(l[prefix:])
>               linecont = l.strip('\n').endswith('\\')
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_mailman_listinfo_mercurial-2Ddevel&d=CwIGaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=mEgSWILcY4c4W3zjApBQLA&m=-tkaXBS16riF5eKtud65I0gpcKvpUpg4aQ0T3soZoj8&s=foAyFrMhEc_4_ny7LnQ237dPamR2E9BTdngueApEd8Q&e=
>
timeless - April 13, 2016, 4:18 p.m.
Simon Farnsworth wrote:
> Apart from the nitpicks in other mails, the series looks OK to me.

Thanks, I'll try to resend.

> Out of interest, how many more import problems does this show in the current
> Mercurial codebase?

I'm not certain. At this point, most modules are affected by a single failure.
It's kind of like "whack-a-mole" on easy -- there's only one hole and you know
the mole will come out once you clean up the mole you just whacked.

You can see my progress here:
https://bitbucket.org/timeless/mercurial-crew/commits/all?search=%3A%3A(desc(pycompat)%2Bdesc(check)%2Bdesc(code)%2Bdesc(py3))
(I don't know how to get bitbucket to search by bookmarks, the
bookmark is py3port)

Patch

diff --git a/contrib/import-checker.py b/contrib/import-checker.py
--- a/contrib/import-checker.py
+++ b/contrib/import-checker.py
@@ -598,19 +598,41 @@ 
     'from __future__ import print_function\\n'
     example[11] doctest.py 11
     'from __future__ import print_function\\\\\\n, absolute_import\\n'
+
+    >>> lines = [
+    ... 'def x():',
+    ... "    ''' # docstring",
+    ... '    x()',
+    ... '    >>> x() + 1',
+    ... '    3',
+    ... "    '''",
+    ... '    return 2',
+    ... ]
+
+    >>> test("example.py", lines)
+    example[4] doctest.py 4
+    'x() + 1\\n'
     """
     inlinepython = 0
     shpython = 0
     script = []
-    prefix = 6
     t = ''
+    pystr = ''
     n = 0
     linecont = 0
+    if f.endswith('.t'):
+        pyinlinestart = b'  >>> '
+        pyinlinecont = b'  ... '
+        prefix = 6
+    else:
+        pyinlinestart = b'>>> '
+        pyinlinecont = b'... '
+        prefix = 4
     for l in src:
         n += 1
         if not l.endswith(b'\n'):
             l += b'\n'
-        if l.startswith(b'  >>> '): # python inlines
+        if l.startswith(pyinlinestart): # python inlines
             if shpython:
                 print("Parse Error %s:%d\n" % (f, n))
             if not inlinepython:
@@ -620,7 +642,7 @@ 
             script.append(l[prefix:])
             linecont = l.strip('\n').endswith('\\')
             continue
-        if l.startswith(b'  ... '): # python inlines
+        if l.startswith(pyinlinecont): # python inlines
             script.append(l[prefix:])
             linecont = l.strip('\n').endswith('\\')
             continue
@@ -652,6 +674,36 @@ 
             inlinepython = 0
             linecont = False
             continue
+        marker = l.lstrip()[:3]
+        if pystr:
+            if marker == pystr:
+                pystr = ''
+                if not inlinepython:
+                    continue
+                yield ''.join(script), ("%s[%d]" %
+                       (modname, inlinepython)), t, inlinepython
+                script = []
+                inlinepython = 0
+                linecont = False
+                continue
+            pydoc = re.search('^(\s*)>>> ', l)
+            if pydoc:
+                pyinlinestart = pydoc.group()
+                pyinlinecont = pyinlinestart.replace('>>>', '...')
+                prefix = len(pyinlinestart)
+                inlinepython = n
+                t = 'doctest.py'
+                script.append(l[prefix:])
+                linecont = l.strip('\n').endswith('\\')
+                continue
+            if pystr == l.strip()[-3:]:
+                pystr = ''
+                continue
+        if marker in ("'''", '"""'):
+            if l.lstrip()[3:].strip().endswith(marker):
+                continue
+            pystr = marker
+            continue
         if linecont:
             script.append(l[prefix:])
             linecont = l.strip('\n').endswith('\\')