Patchwork D5050: tests: add test verifying our Python 3 tokenizer works

login
register
mail settings
Submitter phabricator
Date Oct. 13, 2018, 7:29 a.m.
Message ID <differential-rev-PHID-DREV-64pjrytr5fcsh7l7vkci-req@phab.mercurial-scm.org>
Download mbox | patch
Permalink /patch/35835/
State New
Headers show

Comments

phabricator - Oct. 13, 2018, 7:29 a.m.
indygreg created this revision.
Herald added subscribers: mercurial-devel, mjpieters.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  Our vendored Python 3.7 tokenizer is now fully working on Python 2.7.
  Let's verify that by adding a test that ensures we can tokenize all .py
  files in the repo.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D5050

AFFECTED FILES
  contrib/check-py3-tokenize.py
  contrib/import-checker.py
  tests/test-check-py3-tokenize.t

CHANGE DETAILS




To: indygreg, #hg-reviewers
Cc: mjpieters, mercurial-devel

Patch

diff --git a/tests/test-check-py3-tokenize.t b/tests/test-check-py3-tokenize.t
new file mode 100644
--- /dev/null
+++ b/tests/test-check-py3-tokenize.t
@@ -0,0 +1,8 @@ 
+#require test-repo
+
+  $ . "$TESTDIR/helpers-testrepo.sh"
+  $ cd "$TESTDIR"/..
+
+  $ testrepohg files 'set:**.py' \
+  > 'tests/**.t' \
+  > | sed 's-\\-/-g' | "$PYTHON" contrib/check-py3-tokenize.py -
diff --git a/contrib/import-checker.py b/contrib/import-checker.py
--- a/contrib/import-checker.py
+++ b/contrib/import-checker.py
@@ -24,6 +24,7 @@ 
     '__future__',
     'bzrlib',
     'hgclient',
+    'hgdemandimport',
     'mercurial',
     'mercurial.hgweb.common',
     'mercurial.hgweb.request',
diff --git a/contrib/check-py3-tokenize.py b/contrib/check-py3-tokenize.py
new file mode 100755
--- /dev/null
+++ b/contrib/check-py3-tokenize.py
@@ -0,0 +1,56 @@ 
+#!/usr/bin/env python
+#
+# check-py3-tokenize - check that our Python 3 tokenizer works
+#
+# Copyright Gregory Szorc <gregory.szorc@gmail.com>
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+from __future__ import absolute_import, print_function
+
+import io
+import sys
+
+from hgdemandimport import (
+    py3tokenize,
+)
+
+def checkfile(path):
+    with open(path, 'rb') as fh:
+        b = io.BytesIO(fh.read())
+
+    b.seek(0)
+
+    try:
+        tokens = list(py3tokenize.tokenize(b.readline))
+    except Exception as e:
+        print('%s failed to tokenize: %s' % (path, e))
+        return False
+
+    try:
+        py3tokenize.untokenize(tokens)
+    except Exception as e:
+        print('%s failed to untokenize: %' % (path, e))
+        return False
+
+    return True
+
+if __name__ == '__main__':
+    argv = sys.argv
+
+    if len(argv) < 2 or (argv[1] == '-' and len(argv) > 2):
+        print('Usage: %s {-|file [file] [file] ...}')
+        sys.exit(1)
+
+    if argv[1] == '-':
+        argv = argv[:1]
+        argv.extend(l.rstrip() for l in sys.stdin.readlines())
+
+    code = 0
+
+    for path in argv[1:]:
+        if not checkfile(path):
+            code = 1
+
+    sys.exit(code)