Patchwork py3: handle os.environ.get() case in module loader

login
register
mail settings
Submitter Pulkit Goyal
Date Aug. 4, 2016, 7:11 p.m.
Message ID <6bc900348ed8d695dddc.1470337871@pulkit-goyal>
Download mbox | patch
Permalink /patch/16101/
State Changes Requested
Headers show

Comments

Pulkit Goyal - Aug. 4, 2016, 7:11 p.m.
# HG changeset patch
# User Pulkit Goyal <7895pulkit@gmail.com>
# Date 1470337846 -19800
#      Fri Aug 05 00:40:46 2016 +0530
# Branch stable
# Node ID 6bc900348ed8d695dddc9451f56e3adf38149127
# Parent  44c45c5f2481e94b8eeb24b31fea4b49d3453835
py3: handle os.environ.get() case in module loader

The os.environ.get() doesnot accepts bytes on Python 3. Instead of adding u''
in the source code, its preferred to add a case in the custom module loader
which handles this.

Firstly the code rewrite the token to include u'' to undo the effect by the
transformer on the first argument. Then it searches for more arguments if
present and does the same for them.
Gregory Szorc - Aug. 5, 2016, 3:15 a.m.
On Thu, Aug 4, 2016 at 12:11 PM, Pulkit Goyal <7895pulkit@gmail.com> wrote:

> # HG changeset patch
> # User Pulkit Goyal <7895pulkit@gmail.com>
> # Date 1470337846 -19800
> #      Fri Aug 05 00:40:46 2016 +0530
> # Branch stable
> # Node ID 6bc900348ed8d695dddc9451f56e3adf38149127
> # Parent  44c45c5f2481e94b8eeb24b31fea4b49d3453835
> py3: handle os.environ.get() case in module loader
>
> The os.environ.get() doesnot accepts bytes on Python 3. Instead of adding
> u''
> in the source code, its preferred to add a case in the custom module loader
> which handles this.
>
> Firstly the code rewrite the token to include u'' to undo the effect by the
> transformer on the first argument. Then it searches for more arguments if
> present and does the same for them.
>

I'm not sure if we decided this was the proper course of action. But I'd
rather have a helper that accesses os.environ or os.environb (as
appropriate) and does the appropriate types transform (as appropriate)
rather than hacking more things into the module loader.


>
> diff -r 44c45c5f2481 -r 6bc900348ed8 mercurial/__init__.py
> --- a/mercurial/__init__.py     Thu Aug 04 00:32:19 2016 +0530
> +++ b/mercurial/__init__.py     Fri Aug 05 00:40:46 2016 +0530
> @@ -272,6 +272,47 @@
>                      except IndexError:
>                          pass
>
> +                # os.environ.get() doesnot accepts byte strings on Python
> 3.
> +                # Rewrite the token to include the unicode literal prefix
> so
> +                # the string transformer above doesn't add the byte
> prefix.
> +                if (fn in ('get') and
> +                    prevtoken.type == token.OP and prevtoken.string == '.'
> +                    and tokens[i - 2].string == 'environ' and
> +                    tokens[i - 4].string == 'os'):
> +                    # (tokens[i-4:i-2], os.environ)
> +                    # (OP, '.')
> +                    # (NAME, 'get')
> +                    # (OP, '(')
> +                    # (STRING, 'HGENCODING')
> +                    # (OP, ')')
> +                    try:
> +                        st = tokens[i + 2]
> +                        if (st.type == token.STRING and
> +                            st.string[0] in ("'", '"')):
> +                            rt = tokenize.TokenInfo(st.type, 'u%s' %
> st.string,
> +                                                    st.start, st.end,
> st.line)
> +                            tokens[i + 2] = rt
> +
> +                        # This while loop deals with the further
> arguments in
> +                        # in os.environ.get(). It checks whether we have
> ')'
> +                        # as the next token or we have a ',' which
> indicates
> +                        # that we have more arguments. This loop
> terminates
> +                        # when we get the ')' token.
> +                        j = 3
> +                        while (tokens[i + j].string != ')' and
> +                            tokens[i + j].string == ','):
> +                            st = tokens[i + j + 1]
> +                            if (st.type == token.STRING and
> +                                st.string[0] in ("'", '"')):
> +                                rt = tokenize.TokenInfo(st.type, 'u%s' %
> +                                                        st.string,
> st.start,
> +                                                        st.end, st.line)
> +                                tokens[i + j + 1] = rt
> +                                j += 2
> +
> +                    except IndexError:
> +                        pass
> +
>              # Emit unmodified token.
>              yield t
>
> @@ -279,7 +320,7 @@
>      # ``replacetoken`` or any mechanism that changes semantics of module
>      # loading is changed. Otherwise cached bytecode may get loaded without
>      # the new transformation mechanisms applied.
> -    BYTECODEHEADER = b'HG\x00\x01'
> +    BYTECODEHEADER = b'HG\x00\x02'
>
>      class hgloader(importlib.machinery.SourceFileLoader):
>          """Custom module loader that transforms source code.
> diff -r 44c45c5f2481 -r 6bc900348ed8 tests/test-check-py3-compat.t
> --- a/tests/test-check-py3-compat.t     Thu Aug 04 00:32:19 2016 +0530
> +++ b/tests/test-check-py3-compat.t     Fri Aug 05 00:40:46 2016 +0530
> @@ -15,90 +15,90 @@
>  #if py3exe
>    $ hg files 'set:(**.py)' | sed 's|\\|/|g' | xargs $PYTHON3
> contrib/check-py3-compat.py
>    doc/hgmanpage.py: invalid syntax: invalid syntax (<unknown>, line *)
> (glob)
> -  hgext/acl.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/automv.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/blackbox.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/bugzilla.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/censor.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/chgserver.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/children.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/churn.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/clonebundles.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> +  hgext/acl.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/automv.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/blackbox.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/bugzilla.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/censor.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/chgserver.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/children.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/churn.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/clonebundles.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
>    hgext/color.py: invalid syntax: invalid syntax (<unknown>, line *)
> (glob)
> -  hgext/convert/bzr.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/convert/common.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/convert/convcmd.py: error importing: <TypeError> str expected,
> not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/cvs.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/convert/cvsps.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/convert/darcs.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/convert/filemap.py: error importing: <TypeError> str expected,
> not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/git.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/convert/gnuarch.py: error importing: <TypeError> str expected,
> not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/hg.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/convert/monotone.py: error importing: <TypeError> str expected,
> not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/p4.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/convert/subversion.py: error importing: <TypeError> str expected,
> not bytes (error at encoding.py:*) (glob)
> +  hgext/convert/bzr.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/convert/common.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/convert/convcmd.py: error importing: <TypeError> str expected,
> not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/cvs.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/convert/cvsps.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/convert/darcs.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/convert/filemap.py: error importing: <TypeError> str expected,
> not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/git.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/convert/gnuarch.py: error importing: <TypeError> str expected,
> not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/hg.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/convert/monotone.py: error importing: <TypeError> str expected,
> not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/p4.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/convert/subversion.py: error importing: <TypeError> str expected,
> not bytes (error at i18n.py:*) (glob)
>    hgext/convert/transport.py: error importing module: <ImportError> No
> module named 'svn.client' (line *) (glob)
> -  hgext/eol.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/extdiff.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/factotum.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/fetch.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/fsmonitor/state.py: error importing: <TypeError> str expected,
> not bytes (error at encoding.py:*) (glob)
> -  hgext/fsmonitor/watchmanclient.py: error importing: <TypeError> str
> expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/gpg.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/graphlog.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/hgk.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/highlight/highlight.py: error importing: <TypeError> str
> expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/histedit.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/journal.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/keyword.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/largefiles/basestore.py: error importing: <TypeError> str
> expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/lfcommands.py: error importing: <TypeError> str
> expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/lfutil.py: error importing: <TypeError> str expected,
> not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/localstore.py: error importing: <TypeError> str
> expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/overrides.py: error importing: <TypeError> str
> expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/proto.py: error importing: <TypeError> str expected,
> not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/remotestore.py: error importing: <TypeError> str
> expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/reposetup.py: error importing: <TypeError> str
> expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/storefactory.py: error importing: <TypeError> str
> expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/uisetup.py: error importing: <TypeError> str expected,
> not bytes (error at encoding.py:*) (glob)
> +  hgext/eol.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/extdiff.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/factotum.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/fetch.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/fsmonitor/state.py: error importing: <TypeError> str expected,
> not bytes (error at i18n.py:*) (glob)
> +  hgext/fsmonitor/watchmanclient.py: error importing: <TypeError> str
> expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/gpg.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/graphlog.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/hgk.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/highlight/highlight.py: error importing: <TypeError> str
> expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/histedit.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/journal.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/keyword.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/largefiles/basestore.py: error importing: <TypeError> str
> expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/lfcommands.py: error importing: <TypeError> str
> expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/lfutil.py: error importing: <TypeError> str expected,
> not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/localstore.py: error importing: <TypeError> str
> expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/overrides.py: error importing: <TypeError> str
> expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/proto.py: error importing: <TypeError> str expected,
> not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/remotestore.py: error importing: <TypeError> str
> expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/reposetup.py: error importing: <TypeError> str
> expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/storefactory.py: error importing: <TypeError> str
> expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/uisetup.py: error importing: <TypeError> str expected,
> not bytes (error at i18n.py:*) (glob)
>    hgext/largefiles/wirestore.py: error importing module: <SystemError>
> Parent module 'hgext.largefiles' not loaded, cannot perform relative import
> (line *) (glob)
> -  hgext/mq.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/notify.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/pager.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/patchbomb.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/purge.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/rebase.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/record.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/relink.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/schemes.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/share.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/shelve.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/strip.py: error importing: <TypeError> str expected, not bytes
> (error at encoding.py:*) (glob)
> -  hgext/transplant.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/win32mbcs.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  hgext/win32text.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> +  hgext/mq.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/notify.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/pager.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/patchbomb.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/purge.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/rebase.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/record.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/relink.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/schemes.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/share.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/shelve.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/strip.py: error importing: <TypeError> str expected, not bytes
> (error at i18n.py:*) (glob)
> +  hgext/transplant.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/win32mbcs.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  hgext/win32text.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
>    mercurial/archival.py: invalid syntax: invalid syntax (<unknown>, line
> *) (glob)
> -  mercurial/bookmarks.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/branchmap.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> +  mercurial/bookmarks.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/branchmap.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
>    mercurial/bundle2.py: invalid syntax: invalid syntax (<unknown>, line
> *) (glob)
> -  mercurial/bundlerepo.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/byterange.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/changegroup.py: error importing: <TypeError> str expected,
> not bytes (error at encoding.py:*) (glob)
> -  mercurial/changelog.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/cmdutil.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> +  mercurial/bundlerepo.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/byterange.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/changegroup.py: error importing: <TypeError> str expected,
> not bytes (error at i18n.py:*) (glob)
> +  mercurial/changelog.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/cmdutil.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
>    mercurial/commands.py: invalid syntax: invalid syntax (<unknown>, line
> *) (glob)
> -  mercurial/commandserver.py: error importing: <TypeError> str expected,
> not bytes (error at encoding.py:*) (glob)
> -  mercurial/config.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/context.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/copies.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/crecord.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/dagparser.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/dagutil.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/destutil.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/dirstate.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/discovery.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> -  mercurial/dispatch.py: error importing: <TypeError> str expected, not
> bytes (error at encoding.py:*) (glob)
> +  mercurial/commandserver.py: error importing: <TypeError> str expected,
> not bytes (error at i18n.py:*) (glob)
> +  mercurial/config.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/context.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/copies.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/crecord.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/dagparser.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/dagutil.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/destutil.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/dirstate.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/discovery.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> +  mercurial/dispatch.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
>    mercurial/exchange.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
>    mercurial/extensions.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
>    mercurial/fancyopts.py: error importing: <TypeError> str expected, not
> bytes (error at i18n.py:*) (glob)
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>
Siddharth Agarwal - Aug. 5, 2016, 3:26 a.m.
On 8/4/16 12:11, Pulkit Goyal wrote:
> # HG changeset patch
> # User Pulkit Goyal <7895pulkit@gmail.com>
> # Date 1470337846 -19800
> #      Fri Aug 05 00:40:46 2016 +0530
> # Branch stable
> # Node ID 6bc900348ed8d695dddc9451f56e3adf38149127
> # Parent  44c45c5f2481e94b8eeb24b31fea4b49d3453835
> py3: handle os.environ.get() case in module loader
>
> The os.environ.get() doesnot accepts bytes on Python 3. Instead of adding u''
> in the source code, its preferred to add a case in the custom module loader
> which handles this.

I agree with Greg -- this makes things more complicated than necessary. 
We should just have a helper (e.g. util.environ) that gets assigned to 
os.environ on py2 and os.environb on py3. (And with possibly different 
behavior on Windows, similar to filenames.)

- Siddharth

>
> Firstly the code rewrite the token to include u'' to undo the effect by the
> transformer on the first argument. Then it searches for more arguments if
> present and does the same for them.
>
> diff -r 44c45c5f2481 -r 6bc900348ed8 mercurial/__init__.py
> --- a/mercurial/__init__.py	Thu Aug 04 00:32:19 2016 +0530
> +++ b/mercurial/__init__.py	Fri Aug 05 00:40:46 2016 +0530
> @@ -272,6 +272,47 @@
>                       except IndexError:
>                           pass
>   
> +                # os.environ.get() doesnot accepts byte strings on Python 3.
> +                # Rewrite the token to include the unicode literal prefix so
> +                # the string transformer above doesn't add the byte prefix.
> +                if (fn in ('get') and
> +                    prevtoken.type == token.OP and prevtoken.string == '.'
> +                    and tokens[i - 2].string == 'environ' and
> +                    tokens[i - 4].string == 'os'):
> +                    # (tokens[i-4:i-2], os.environ)
> +                    # (OP, '.')
> +                    # (NAME, 'get')
> +                    # (OP, '(')
> +                    # (STRING, 'HGENCODING')
> +                    # (OP, ')')
> +                    try:
> +                        st = tokens[i + 2]
> +                        if (st.type == token.STRING and
> +                            st.string[0] in ("'", '"')):
> +                            rt = tokenize.TokenInfo(st.type, 'u%s' % st.string,
> +                                                    st.start, st.end, st.line)
> +                            tokens[i + 2] = rt
> +
> +                        # This while loop deals with the further arguments in
> +                        # in os.environ.get(). It checks whether we have ')'
> +                        # as the next token or we have a ',' which indicates
> +                        # that we have more arguments. This loop terminates
> +                        # when we get the ')' token.
> +                        j = 3
> +                        while (tokens[i + j].string != ')' and
> +                            tokens[i + j].string == ','):
> +                            st = tokens[i + j + 1]
> +                            if (st.type == token.STRING and
> +                                st.string[0] in ("'", '"')):
> +                                rt = tokenize.TokenInfo(st.type, 'u%s' %
> +                                                        st.string, st.start,
> +                                                        st.end, st.line)
> +                                tokens[i + j + 1] = rt
> +                                j += 2
> +
> +                    except IndexError:
> +                        pass
> +
>               # Emit unmodified token.
>               yield t
>   
> @@ -279,7 +320,7 @@
>       # ``replacetoken`` or any mechanism that changes semantics of module
>       # loading is changed. Otherwise cached bytecode may get loaded without
>       # the new transformation mechanisms applied.
> -    BYTECODEHEADER = b'HG\x00\x01'
> +    BYTECODEHEADER = b'HG\x00\x02'
>   
>       class hgloader(importlib.machinery.SourceFileLoader):
>           """Custom module loader that transforms source code.
> diff -r 44c45c5f2481 -r 6bc900348ed8 tests/test-check-py3-compat.t
> --- a/tests/test-check-py3-compat.t	Thu Aug 04 00:32:19 2016 +0530
> +++ b/tests/test-check-py3-compat.t	Fri Aug 05 00:40:46 2016 +0530
> @@ -15,90 +15,90 @@
>   #if py3exe
>     $ hg files 'set:(**.py)' | sed 's|\\|/|g' | xargs $PYTHON3 contrib/check-py3-compat.py
>     doc/hgmanpage.py: invalid syntax: invalid syntax (<unknown>, line *) (glob)
> -  hgext/acl.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/automv.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/blackbox.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/bugzilla.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/censor.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/chgserver.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/children.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/churn.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/clonebundles.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> +  hgext/acl.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/automv.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/blackbox.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/bugzilla.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/censor.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/chgserver.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/children.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/churn.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/clonebundles.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
>     hgext/color.py: invalid syntax: invalid syntax (<unknown>, line *) (glob)
> -  hgext/convert/bzr.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/common.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/convcmd.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/cvs.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/cvsps.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/darcs.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/filemap.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/git.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/gnuarch.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/hg.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/monotone.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/p4.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/convert/subversion.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> +  hgext/convert/bzr.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/common.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/convcmd.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/cvs.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/cvsps.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/darcs.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/filemap.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/git.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/gnuarch.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/hg.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/monotone.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/p4.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/convert/subversion.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
>     hgext/convert/transport.py: error importing module: <ImportError> No module named 'svn.client' (line *) (glob)
> -  hgext/eol.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/extdiff.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/factotum.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/fetch.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/fsmonitor/state.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/fsmonitor/watchmanclient.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/gpg.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/graphlog.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/hgk.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/highlight/highlight.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/histedit.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/journal.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/keyword.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/basestore.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/lfcommands.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/lfutil.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/localstore.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/overrides.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/proto.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/remotestore.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/reposetup.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/storefactory.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/largefiles/uisetup.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> +  hgext/eol.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/extdiff.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/factotum.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/fetch.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/fsmonitor/state.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/fsmonitor/watchmanclient.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/gpg.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/graphlog.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/hgk.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/highlight/highlight.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/histedit.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/journal.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/keyword.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/basestore.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/lfcommands.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/lfutil.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/localstore.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/overrides.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/proto.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/remotestore.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/reposetup.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/storefactory.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/largefiles/uisetup.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
>     hgext/largefiles/wirestore.py: error importing module: <SystemError> Parent module 'hgext.largefiles' not loaded, cannot perform relative import (line *) (glob)
> -  hgext/mq.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/notify.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/pager.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/patchbomb.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/purge.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/rebase.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/record.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/relink.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/schemes.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/share.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/shelve.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/strip.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/transplant.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/win32mbcs.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  hgext/win32text.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> +  hgext/mq.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/notify.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/pager.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/patchbomb.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/purge.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/rebase.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/record.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/relink.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/schemes.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/share.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/shelve.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/strip.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/transplant.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/win32mbcs.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  hgext/win32text.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
>     mercurial/archival.py: invalid syntax: invalid syntax (<unknown>, line *) (glob)
> -  mercurial/bookmarks.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/branchmap.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> +  mercurial/bookmarks.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/branchmap.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
>     mercurial/bundle2.py: invalid syntax: invalid syntax (<unknown>, line *) (glob)
> -  mercurial/bundlerepo.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/byterange.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/changegroup.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/changelog.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/cmdutil.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> +  mercurial/bundlerepo.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/byterange.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/changegroup.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/changelog.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/cmdutil.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
>     mercurial/commands.py: invalid syntax: invalid syntax (<unknown>, line *) (glob)
> -  mercurial/commandserver.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/config.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/context.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/copies.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/crecord.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/dagparser.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/dagutil.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/destutil.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/dirstate.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/discovery.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> -  mercurial/dispatch.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
> +  mercurial/commandserver.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/config.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/context.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/copies.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/crecord.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/dagparser.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/dagutil.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/destutil.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/dirstate.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/discovery.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> +  mercurial/dispatch.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
>     mercurial/exchange.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
>     mercurial/extensions.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
>     mercurial/fancyopts.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Jun Wu - Aug. 5, 2016, 11:12 a.m.
util.environ looks better. ui.environ has issues for places without the ui
object.

I suggested change the transformer in the previous py3 thread because I felt
that was better than changing individual files manually.

Excerpts from Siddharth Agarwal's message of 2016-08-04 20:26:54 -0700:
> I agree with Greg -- this makes things more complicated than necessary. 
> We should just have a helper (e.g. util.environ) that gets assigned to 
> os.environ on py2 and os.environb on py3. (And with possibly different 
> behavior on Windows, similar to filenames.)
> 
> - Siddharth
Katsunori FUJIWARA - Aug. 5, 2016, 4:18 p.m.
At Fri, 5 Aug 2016 12:12:36 +0100,
Jun Wu wrote:
> 
> util.environ looks better. ui.environ has issues for places without the ui
> object.

IMHO, for availability (and less code duplication), we should define
such helpers in other than util.py, because using helpers in util.py
causes cyclic importing in i18n, encoding and so on.

Of course, making util.py provide imported helpers as own functions
can avoid adding new import statement for such helper.

The role of pycompat.py itself seems good for this purpose, but is it
a little large to be imported in i18n, encoding and so on. ?


> I suggested change the transformer in the previous py3 thread because I felt
> that was better than changing individual files manually.
> 
> Excerpts from Siddharth Agarwal's message of 2016-08-04 20:26:54 -0700:
> > I agree with Greg -- this makes things more complicated than necessary. 
> > We should just have a helper (e.g. util.environ) that gets assigned to 
> > os.environ on py2 and os.environb on py3. (And with possibly different 
> > behavior on Windows, similar to filenames.)
> > 
> > - Siddharth
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel

----------------------------------------------------------------------
[FUJIWARA Katsunori]                             foozy@lares.dti.ne.jp
Yuya Nishihara - Aug. 6, 2016, 3:08 a.m.
On Sat, 06 Aug 2016 01:18:05 +0900, FUJIWARA Katsunori wrote:
> At Fri, 5 Aug 2016 12:12:36 +0100,
> Jun Wu wrote:
> > 
> > util.environ looks better. ui.environ has issues for places without the ui
> > object.
> 
> IMHO, for availability (and less code duplication), we should define
> such helpers in other than util.py, because using helpers in util.py
> causes cyclic importing in i18n, encoding and so on.
> 
> Of course, making util.py provide imported helpers as own functions
> can avoid adding new import statement for such helper.
> 
> The role of pycompat.py itself seems good for this purpose, but is it
> a little large to be imported in i18n, encoding and so on. ?

pycompat seems fine. It should be loaded anyway on Python 3 to reintroduce
xrange().
Pierre-Yves David - Aug. 6, 2016, 3:14 a.m.
On 08/06/2016 05:08 AM, Yuya Nishihara wrote:
> On Sat, 06 Aug 2016 01:18:05 +0900, FUJIWARA Katsunori wrote:
>> At Fri, 5 Aug 2016 12:12:36 +0100,
>> Jun Wu wrote:
>>>
>>> util.environ looks better. ui.environ has issues for places without the ui
>>> object.
>>
>> IMHO, for availability (and less code duplication), we should define
>> such helpers in other than util.py, because using helpers in util.py
>> causes cyclic importing in i18n, encoding and so on.
>>
>> Of course, making util.py provide imported helpers as own functions
>> can avoid adding new import statement for such helper.
>>
>> The role of pycompat.py itself seems good for this purpose, but is it
>> a little large to be imported in i18n, encoding and so on. ?
>
> pycompat seems fine. It should be loaded anyway on Python 3 to reintroduce
> xrange().

+1 for pycompat
Siddharth Agarwal - Aug. 17, 2016, 5:04 p.m.
On 8/4/16 20:26, Siddharth Agarwal wrote:
>
> I agree with Greg -- this makes things more complicated than 
> necessary. We should just have a helper (e.g. util.environ) that gets 
> assigned to os.environ on py2 and os.environb on py3. (And with 
> possibly different behavior on Windows, similar to filenames.)

Pulkit asked me to elaborate a bit:

For Python 3 on Unix (including OS X), byte strings (UTF-8 encoded byte 
strings on OS X) are as high fidelity a way to talk to the native OS 
APIs as Unicode strings. So on Unix, you don't lose any information by 
using os.environb versus os.environ.

For Python 3 on Windows, byte strings are a *lower* fidelity way to talk 
to the native OS APIs than Unicode strings. So os.environb gives you 
potentially less information than os.environ.

This behavior is identical to the way filesystem APIs work on Windows. 
See https://www.mercurial-scm.org/wiki/WindowsUTF8Plan for how Mercurial 
has tackled/plans to tackle this problem.

- Siddharth
Gregory Szorc - Aug. 17, 2016, 5:21 p.m.
On Wed, Aug 17, 2016 at 10:04 AM, Siddharth Agarwal <sid@less-broken.com>
wrote:

> On 8/4/16 20:26, Siddharth Agarwal wrote:
>
>>
>> I agree with Greg -- this makes things more complicated than necessary.
>> We should just have a helper (e.g. util.environ) that gets assigned to
>> os.environ on py2 and os.environb on py3. (And with possibly different
>> behavior on Windows, similar to filenames.)
>>
>
> Pulkit asked me to elaborate a bit:
>
> For Python 3 on Unix (including OS X), byte strings (UTF-8 encoded byte
> strings on OS X) are as high fidelity a way to talk to the native OS APIs
> as Unicode strings. So on Unix, you don't lose any information by using
> os.environb versus os.environ.
>
> For Python 3 on Windows, byte strings are a *lower* fidelity way to talk
> to the native OS APIs than Unicode strings. So os.environb gives you
> potentially less information than os.environ.
>
> This behavior is identical to the way filesystem APIs work on Windows. See
> https://www.mercurial-scm.org/wiki/WindowsUTF8Plan for how Mercurial has
> tackled/plans to tackle this problem.


WindowsUTF8Plan has existed for years without any movement. Addressing that
will require a massive refactor. Since it isn't on our radar, I think it is
out of scope for Python 3 porting work. Can we just use os.environb/bytes
paths on Python 3 for now and deal with Windows paths compatibility another
time? Worst case this doesn't make Mercurial usable under Python 3 on
Windows. I'm not sure anyone will care until Mercurial actually works with
Python 3. And that's a way off. I think I'm fine kicking the can down the
road to unblock the overall Python 3 porting effort.
Siddharth Agarwal - Aug. 17, 2016, 5:46 p.m.
On 8/17/16 10:21, Gregory Szorc wrote:
> WindowsUTF8Plan has existed for years without any movement. Addressing 
> that will require a massive refactor. Since it isn't on our radar, I 
> think it is out of scope for Python 3 porting work. Can we just use 
> os.environb/bytes paths on Python 3 for now and deal with Windows 
> paths compatibility another time? Worst case this doesn't make 
> Mercurial usable under Python 3 on Windows. I'm not sure anyone will 
> care until Mercurial actually works with Python 3. And that's a way 
> off. I think I'm fine kicking the can down the road to unblock the 
> overall Python 3 porting effort.


Sure, that's fine by me (and it's not like our current behavior with 
Python 2 is optimal either). I just wanted to flag this as a potential 
area of concern.
Yuya Nishihara - Aug. 18, 2016, 8:29 a.m.
On Wed, 17 Aug 2016 10:46:01 -0700, Siddharth Agarwal wrote:
> On 8/17/16 10:21, Gregory Szorc wrote:
> > WindowsUTF8Plan has existed for years without any movement. Addressing 
> > that will require a massive refactor. Since it isn't on our radar, I 
> > think it is out of scope for Python 3 porting work. Can we just use 
> > os.environb/bytes paths on Python 3 for now and deal with Windows 
> > paths compatibility another time? Worst case this doesn't make 
> > Mercurial usable under Python 3 on Windows. I'm not sure anyone will 
> > care until Mercurial actually works with Python 3. And that's a way 
> > off. I think I'm fine kicking the can down the road to unblock the 
> > overall Python 3 porting effort.
> 
> Sure, that's fine by me (and it's not like our current behavior with 
> Python 2 is optimal either). I just wanted to flag this as a potential 
> area of concern.

Anyway, we can get back unicodes from our environb wrapper which would
probably store localstr objects, and pass them to subprocess.
Pulkit Goyal - Sept. 3, 2016, 3:44 p.m.
os.environ for py2 and os.environb for py3, in that case rather having
util.envrion, it will be better to convert environ to environb using
the transformer. What say?

On Thu, Aug 18, 2016 at 1:59 PM, Yuya Nishihara <yuya@tcha.org> wrote:
> On Wed, 17 Aug 2016 10:46:01 -0700, Siddharth Agarwal wrote:
>> On 8/17/16 10:21, Gregory Szorc wrote:
>> > WindowsUTF8Plan has existed for years without any movement. Addressing
>> > that will require a massive refactor. Since it isn't on our radar, I
>> > think it is out of scope for Python 3 porting work. Can we just use
>> > os.environb/bytes paths on Python 3 for now and deal with Windows
>> > paths compatibility another time? Worst case this doesn't make
>> > Mercurial usable under Python 3 on Windows. I'm not sure anyone will
>> > care until Mercurial actually works with Python 3. And that's a way
>> > off. I think I'm fine kicking the can down the road to unblock the
>> > overall Python 3 porting effort.
>>
>> Sure, that's fine by me (and it's not like our current behavior with
>> Python 2 is optimal either). I just wanted to flag this as a potential
>> area of concern.
>
> Anyway, we can get back unicodes from our environb wrapper which would
> probably store localstr objects, and pass them to subprocess.
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Yuya Nishihara - Sept. 4, 2016, 10:43 a.m.
On Sat, 3 Sep 2016 21:14:37 +0530, Pulkit Goyal wrote:
> os.environ for py2 and os.environb for py3, in that case rather having
> util.envrion, it will be better to convert environ to environb using
> the transformer. What say?

We'll need util.environ or pycompat.environ anyway because there is Windows.

  # pycompat.py
  if py2:
      environ = os.environ
  else:
      try:
          environ = os.environb
      except AttributeError:  # Windows
          environ = wrapper(os.environ)

IMO, pycompat.environ should be read-only to keep the wrapper simple.

Patch

diff -r 44c45c5f2481 -r 6bc900348ed8 mercurial/__init__.py
--- a/mercurial/__init__.py	Thu Aug 04 00:32:19 2016 +0530
+++ b/mercurial/__init__.py	Fri Aug 05 00:40:46 2016 +0530
@@ -272,6 +272,47 @@ 
                     except IndexError:
                         pass
 
+                # os.environ.get() doesnot accepts byte strings on Python 3.
+                # Rewrite the token to include the unicode literal prefix so
+                # the string transformer above doesn't add the byte prefix.
+                if (fn in ('get') and
+                    prevtoken.type == token.OP and prevtoken.string == '.'
+                    and tokens[i - 2].string == 'environ' and
+                    tokens[i - 4].string == 'os'):
+                    # (tokens[i-4:i-2], os.environ)
+                    # (OP, '.')
+                    # (NAME, 'get')
+                    # (OP, '(')
+                    # (STRING, 'HGENCODING')
+                    # (OP, ')')
+                    try:
+                        st = tokens[i + 2]
+                        if (st.type == token.STRING and
+                            st.string[0] in ("'", '"')):
+                            rt = tokenize.TokenInfo(st.type, 'u%s' % st.string,
+                                                    st.start, st.end, st.line)
+                            tokens[i + 2] = rt
+
+                        # This while loop deals with the further arguments in
+                        # in os.environ.get(). It checks whether we have ')'
+                        # as the next token or we have a ',' which indicates
+                        # that we have more arguments. This loop terminates
+                        # when we get the ')' token.
+                        j = 3
+                        while (tokens[i + j].string != ')' and
+                            tokens[i + j].string == ','):
+                            st = tokens[i + j + 1]
+                            if (st.type == token.STRING and
+                                st.string[0] in ("'", '"')):
+                                rt = tokenize.TokenInfo(st.type, 'u%s' %
+                                                        st.string, st.start,
+                                                        st.end, st.line)
+                                tokens[i + j + 1] = rt
+                                j += 2
+
+                    except IndexError:
+                        pass
+
             # Emit unmodified token.
             yield t
 
@@ -279,7 +320,7 @@ 
     # ``replacetoken`` or any mechanism that changes semantics of module
     # loading is changed. Otherwise cached bytecode may get loaded without
     # the new transformation mechanisms applied.
-    BYTECODEHEADER = b'HG\x00\x01'
+    BYTECODEHEADER = b'HG\x00\x02'
 
     class hgloader(importlib.machinery.SourceFileLoader):
         """Custom module loader that transforms source code.
diff -r 44c45c5f2481 -r 6bc900348ed8 tests/test-check-py3-compat.t
--- a/tests/test-check-py3-compat.t	Thu Aug 04 00:32:19 2016 +0530
+++ b/tests/test-check-py3-compat.t	Fri Aug 05 00:40:46 2016 +0530
@@ -15,90 +15,90 @@ 
 #if py3exe
   $ hg files 'set:(**.py)' | sed 's|\\|/|g' | xargs $PYTHON3 contrib/check-py3-compat.py
   doc/hgmanpage.py: invalid syntax: invalid syntax (<unknown>, line *) (glob)
-  hgext/acl.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/automv.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/blackbox.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/bugzilla.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/censor.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/chgserver.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/children.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/churn.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/clonebundles.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
+  hgext/acl.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/automv.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/blackbox.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/bugzilla.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/censor.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/chgserver.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/children.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/churn.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/clonebundles.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
   hgext/color.py: invalid syntax: invalid syntax (<unknown>, line *) (glob)
-  hgext/convert/bzr.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/common.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/convcmd.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/cvs.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/cvsps.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/darcs.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/filemap.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/git.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/gnuarch.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/hg.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/monotone.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/p4.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/convert/subversion.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
+  hgext/convert/bzr.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/common.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/convcmd.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/cvs.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/cvsps.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/darcs.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/filemap.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/git.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/gnuarch.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/hg.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/monotone.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/p4.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/convert/subversion.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
   hgext/convert/transport.py: error importing module: <ImportError> No module named 'svn.client' (line *) (glob)
-  hgext/eol.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/extdiff.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/factotum.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/fetch.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/fsmonitor/state.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/fsmonitor/watchmanclient.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/gpg.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/graphlog.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/hgk.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/highlight/highlight.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/histedit.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/journal.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/keyword.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/largefiles/basestore.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/largefiles/lfcommands.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/largefiles/lfutil.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/largefiles/localstore.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/largefiles/overrides.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/largefiles/proto.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/largefiles/remotestore.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/largefiles/reposetup.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/largefiles/storefactory.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/largefiles/uisetup.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
+  hgext/eol.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/extdiff.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/factotum.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/fetch.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/fsmonitor/state.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/fsmonitor/watchmanclient.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/gpg.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/graphlog.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/hgk.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/highlight/highlight.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/histedit.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/journal.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/keyword.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/largefiles/basestore.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/largefiles/lfcommands.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/largefiles/lfutil.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/largefiles/localstore.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/largefiles/overrides.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/largefiles/proto.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/largefiles/remotestore.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/largefiles/reposetup.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/largefiles/storefactory.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/largefiles/uisetup.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
   hgext/largefiles/wirestore.py: error importing module: <SystemError> Parent module 'hgext.largefiles' not loaded, cannot perform relative import (line *) (glob)
-  hgext/mq.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/notify.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/pager.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/patchbomb.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/purge.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/rebase.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/record.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/relink.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/schemes.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/share.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/shelve.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/strip.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/transplant.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/win32mbcs.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  hgext/win32text.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
+  hgext/mq.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/notify.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/pager.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/patchbomb.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/purge.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/rebase.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/record.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/relink.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/schemes.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/share.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/shelve.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/strip.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/transplant.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/win32mbcs.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  hgext/win32text.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
   mercurial/archival.py: invalid syntax: invalid syntax (<unknown>, line *) (glob)
-  mercurial/bookmarks.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/branchmap.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
+  mercurial/bookmarks.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/branchmap.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
   mercurial/bundle2.py: invalid syntax: invalid syntax (<unknown>, line *) (glob)
-  mercurial/bundlerepo.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/byterange.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/changegroup.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/changelog.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/cmdutil.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
+  mercurial/bundlerepo.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/byterange.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/changegroup.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/changelog.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/cmdutil.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
   mercurial/commands.py: invalid syntax: invalid syntax (<unknown>, line *) (glob)
-  mercurial/commandserver.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/config.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/context.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/copies.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/crecord.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/dagparser.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/dagutil.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/destutil.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/dirstate.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/discovery.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
-  mercurial/dispatch.py: error importing: <TypeError> str expected, not bytes (error at encoding.py:*) (glob)
+  mercurial/commandserver.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/config.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/context.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/copies.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/crecord.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/dagparser.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/dagutil.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/destutil.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/dirstate.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/discovery.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
+  mercurial/dispatch.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
   mercurial/exchange.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
   mercurial/extensions.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)
   mercurial/fancyopts.py: error importing: <TypeError> str expected, not bytes (error at i18n.py:*) (glob)