Patchwork [1,of,3,V2] rust: module policy with importrust

login
register
mail settings
Submitter Georges Racinet
Date June 12, 2019, 8:59 a.m.
Message ID <8ce3da13cda32ee6a1f7.1560329990@ishtar>
Download mbox | patch
Permalink /patch/40440/
State Accepted
Headers show

Comments

Georges Racinet - June 12, 2019, 8:59 a.m.
# HG changeset patch
# User Georges Racinet <georges.racinet@octobus.net>
# Date 1559129276 -7200
#      Wed May 29 13:27:56 2019 +0200
# Node ID 8ce3da13cda32ee6a1f7b4458aa9948e6ef2df8f
# Parent  aae93201f75888deac664d16e01af3c4bc76ac22
# EXP-Topic rust-modulepolicy
rust: module policy with importrust

We introduce two rust+c module policies and a new
`policy.importrust()` that makes use of them.

This simple approach provides runtime switching of
implementations, which is crucial for the performance
measurements such as those Octobus does with ASV.

It can also be useful for bug analysis.

It also has the advantage of making conditionals in
Rust callers more uniform, in particular
abstracting over specifics like `demandimport`

At this point, the build stays unchanged, with the rust-cpython based
`rustext` module being built if HGWITHRUSTEXT=cpython.

More transparency for the callers, i.e., just using
`policy.importmod` would be a much longer term and riskier
effort for the following reasons:

1. It would require to define common module boundaries
   for the three or four cases (pure, c, rust+ext, cffi) and that
   is premature with the Rust extension currently under heavy
   development in areas that are outside the scope of the C extensions.
2. It would imply internal API changes that are not currently wished,
   as the case of ancestors demonstrates.
3. The lack of data or property-like attributes (tp_member
   and tp_getset) in current `rust-cpython` makes it impossible to
   achieve direct transparent replacement of pure Python classes by
   Rust extension code, meaning that the caller sometimes has to be able
   to make adjustments or provide additional wrapping.
Yuya Nishihara - June 12, 2019, 11:28 p.m.
On Wed, 12 Jun 2019 09:59:50 +0100, Georges Racinet wrote:
> # HG changeset patch
> # User Georges Racinet <georges.racinet@octobus.net>
> # Date 1559129276 -7200
> #      Wed May 29 13:27:56 2019 +0200
> # Node ID 8ce3da13cda32ee6a1f7b4458aa9948e6ef2df8f
> # Parent  aae93201f75888deac664d16e01af3c4bc76ac22
> # EXP-Topic rust-modulepolicy
> rust: module policy with importrust

Queued, thanks!

Patch

diff -r aae93201f758 -r 8ce3da13cda3 mercurial/debugcommands.py
--- a/mercurial/debugcommands.py	Fri May 31 22:38:04 2019 -0700
+++ b/mercurial/debugcommands.py	Wed May 29 13:27:56 2019 +0200
@@ -1278,16 +1278,28 @@ 
     fm.write('hgmodules', _("checking installed modules (%s)...\n"),
              os.path.dirname(pycompat.fsencode(__file__)))
 
-    if policy.policy in ('c', 'allow'):
+    rustandc = policy.policy in ('rust+c', 'rust+c-allow')
+    rustext = rustandc  # for now, that's the only case
+    cext = policy.policy in ('c', 'allow') or rustandc
+    nopure = cext or rustext
+    if nopure:
         err = None
         try:
-            from .cext import (
-                base85,
-                bdiff,
-                mpatch,
-                osutil,
-            )
-            dir(bdiff), dir(mpatch), dir(base85), dir(osutil) # quiet pyflakes
+            if cext:
+                from .cext import (
+                    base85,
+                    bdiff,
+                    mpatch,
+                    osutil,
+                )
+                # quiet pyflakes
+                dir(bdiff), dir(mpatch), dir(base85), dir(osutil)
+            if rustext:
+                from .rustext import (
+                    ancestor,
+                    dirstate,
+                )
+                dir(ancestor), dir(dirstate) # quiet pyflakes
         except Exception as inst:
             err = stringutil.forcebytestr(inst)
             problems += 1
diff -r aae93201f758 -r 8ce3da13cda3 mercurial/policy.py
--- a/mercurial/policy.py	Fri May 31 22:38:04 2019 -0700
+++ b/mercurial/policy.py	Wed May 29 13:27:56 2019 +0200
@@ -13,6 +13,9 @@ 
 # Rules for how modules can be loaded. Values are:
 #
 #    c - require C extensions
+#    rust+c - require Rust and C extensions
+#    rust+c-allow - allow Rust and C extensions with fallback to pure Python
+#                   for each
 #    allow - allow pure Python implementation when C loading fails
 #    cffi - required cffi versions (implemented within pure module)
 #    cffi-allow - allow pure Python implementation if cffi version is missing
@@ -29,6 +32,9 @@ 
     b'cffi': (r'cffi', None),
     b'cffi-allow': (r'cffi', r'pure'),
     b'py': (None, r'pure'),
+    # For now, rust policies impact importrust only
+    b'rust+c': (r'cext', None),
+    b'rust+c-allow': (r'cext', r'pure'),
 }
 
 try:
@@ -107,3 +113,34 @@ 
                 raise
     pn, mn = _modredirects.get((purepkg, modname), (purepkg, modname))
     return _importfrom(pn, mn)
+
+def _isrustpermissive():
+    """Assuming the policy is a Rust one, tell if it's permissive."""
+    return policy.endswith(b'-allow')
+
+def importrust(modname, member=None, default=None):
+    """Import Rust module according to policy and availability.
+
+    If policy isn't a Rust one, this returns `default`.
+
+    If either the module or its member is non available, this returns `default`
+    if policy is permissive and raises `ImportError` if not.
+    """
+    if not policy.startswith(b'rust'):
+        return default
+
+    try:
+        mod = _importfrom(r'rustext', modname)
+    except ImportError:
+        if _isrustpermissive():
+            return default
+        raise
+    if member is None:
+        return mod
+
+    try:
+        return getattr(mod, member)
+    except AttributeError:
+        if _isrustpermissive():
+            return default
+        raise ImportError(r"Cannot import name %s" % member)
diff -r aae93201f758 -r 8ce3da13cda3 setup.py
--- a/setup.py	Fri May 31 22:38:04 2019 -0700
+++ b/setup.py	Wed May 29 13:27:56 2019 +0200
@@ -556,10 +556,17 @@ 
         if self.distribution.pure:
             modulepolicy = 'py'
         elif self.build_lib == '.':
-            # in-place build should run without rebuilding C extensions
-            modulepolicy = 'allow'
+            # in-place build should run without rebuilding C
+            # and Rust extensions
+            if hgrustext == 'cpython':
+                modulepolicy = 'rust+c-allow'
+            else:
+                modulepolicy = 'allow'
         else:
-            modulepolicy = 'c'
+            if hgrustext == 'cpython':
+                modulepolicy = 'rust+c'
+            else:
+                modulepolicy = 'c'
 
         content = b''.join([
             b'# this file is autogenerated by setup.py\n',