Patchwork D10688: packaging: use PyOxidizer for producing WiX MSI installer

login
register
mail settings
Submitter phabricator
Date May 7, 2021, 12:01 a.m.
Message ID <differential-rev-PHID-DREV-dhfo4vsutsnsevtmm6ut-req@mercurial-scm.org>
Download mbox | patch
Permalink /patch/49007/
State Superseded
Headers show

Comments

phabricator - May 7, 2021, 12:01 a.m.
indygreg created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  We recently taught our in-tree PyOxidizer configuration file to
  produce MSI installers with WiX using PyOxidizer's built-in
  support for doing so.
  
  This commit changes our WiX + PyOxidizer installer generation
  code to use this functionality.
  
  After this change, all the Python packaging code is doing is the
  following:
  
  - Building HTML documentation
  - Making gettext available to the build process.
  - Munging CLI arguments to variables for the `pyoxidizer` execution.
  - Invoking `pyoxidizer build`.
  - Copying the produced `.msi` to the `dist/` directory.
  
  Applying this stack on stable and rebuilding the 5.8 MSI installer
  produced the following differences from the official 5.8 installer:
  
  - .exe and .pyd files aren't byte identical (this is expected).
  - Various .dist-info/ directories have different names due to older versions of PyOxidizer being buggy and not properly normalizing package names. (The new behavior is correct.)
  - Various *.dist-info/RECORD files are different due to content divergence of files (this is expected).
  - The python38.dll differs due to newer PyOxidizer shipping a newer version of Python 3.8.
  - We now ship python3.dll because PyOxidizer now includes this file by default.
  - The vcruntime140.dll differs because newer PyOxidizer installs a newer version. We also now ship a vcruntime140_1.dll because newer versions of the redistributable ship 2 files now.
  
  The WiX GUIDs and IDs of installed files have likely changed as a
  result of PyOxidizer's different mechanism for generating those
  identifiers. This means that an upgrade install of the MSI will
  replace files instead of doing an incremental update. This is
  likely harmless and we've incurred this kind of breakage before.
  
  As far as I can tell, the new PyOxidizer-built MSI is functionally
  equivalent to the old method. Once we drop support for Python 2.7
  MSI installers, we can delete the WiX code from the repository.
  
  This commit temporarily drops support for extra `.wxs` files. We
  raise an exception instead of silently not using them, which I think
  is appropriate. We should be able to add support back in by injecting
  state into pyoxidizer.bzl via `--var`. I just didn't want to expend
  cognitive load to think about the solution as part of this series.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D10688

AFFECTED FILES
  contrib/packaging/hgpackaging/pyoxidizer.py
  contrib/packaging/hgpackaging/wix.py

CHANGE DETAILS




To: indygreg, #hg-reviewers
Cc: mercurial-patches, mercurial-devel

Patch

diff --git a/contrib/packaging/hgpackaging/wix.py b/contrib/packaging/hgpackaging/wix.py
--- a/contrib/packaging/hgpackaging/wix.py
+++ b/contrib/packaging/hgpackaging/wix.py
@@ -22,7 +22,11 @@ 
     build_py2exe,
     stage_install,
 )
-from .pyoxidizer import create_pyoxidizer_install_layout
+from .pyoxidizer import (
+    build_docs_html,
+    create_pyoxidizer_install_layout,
+    run_pyoxidizer,
+)
 from .util import (
     extract_zip_to_directory,
     normalize_windows_version,
@@ -386,37 +390,65 @@ 
     """Build a WiX MSI installer using PyOxidizer."""
     hg_build_dir = source_dir / "build"
     build_dir = hg_build_dir / ("wix-%s" % target_triple)
-    staging_dir = build_dir / "stage"
-
-    arch = "x64" if "x86_64" in target_triple else "x86"
 
     build_dir.mkdir(parents=True, exist_ok=True)
-    create_pyoxidizer_install_layout(
-        source_dir, build_dir, staging_dir, target_triple
+
+    # Need to ensure docs HTML is built because this isn't done as part of
+    # `pip install Mercurial`.
+    build_docs_html(source_dir)
+
+    build_vars = {}
+
+    if msi_name:
+        build_vars["MSI_NAME"] = msi_name
+
+    if version:
+        build_vars["VERSION"] = version
+
+    if extra_features:
+        build_vars["EXTRA_MSI_FEATURES"] = ";".join(extra_features)
+
+    if signing_info:
+        if signing_info["cert_path"]:
+            build_vars["SIGNING_PFX_PATH"] = signing_info["cert_path"]
+        if signing_info["cert_password"]:
+            build_vars["SIGNING_PFX_PASSWORD"] = signing_info["cert_password"]
+        if signing_info["subject_name"]:
+            build_vars["SIGNING_SUBJECT_NAME"] = signing_info["subject_name"]
+        if signing_info["timestamp_url"]:
+            build_vars["TIME_STAMP_SERVER_URL"] = signing_info["timestamp_url"]
+
+    if extra_wxs:
+        raise Exception(
+            "support for extra .wxs files has been temporarily dropped"
+        )
+
+    out_dir = run_pyoxidizer(
+        source_dir,
+        build_dir,
+        target_triple,
+        build_vars=build_vars,
+        target="msi",
     )
 
-    # We also install some extra files.
-    process_install_rules(EXTRA_INSTALL_RULES, source_dir, staging_dir)
+    msi_dir = out_dir / "msi"
+    msi_files = [f for f in os.listdir(msi_dir) if f.endswith(".msi")]
 
-    # And remove some files we don't want.
-    for f in STAGING_REMOVE_FILES:
-        p = staging_dir / f
-        if p.exists():
-            print('removing %s' % p)
-            p.unlink()
+    if len(msi_files) != 1:
+        raise Exception("expected exactly 1 .msi file; got %d" % len(msi_files))
+
+    msi_filename = msi_files[0]
 
-    return run_wix_packaging(
-        source_dir,
-        build_dir,
-        staging_dir,
-        arch,
-        version,
-        python2=False,
-        msi_name=msi_name,
-        extra_wxs=extra_wxs,
-        extra_features=extra_features,
-        signing_info=signing_info,
-    )
+    msi_path = msi_dir / msi_filename
+    dist_path = source_dir / "dist" / msi_filename
+
+    dist_path.parent.mkdir(parents=True, exist_ok=True)
+
+    shutil.copyfile(msi_path, dist_path)
+
+    return {
+        "msi_path": dist_path,
+    }
 
 
 def run_wix_packaging(
diff --git a/contrib/packaging/hgpackaging/pyoxidizer.py b/contrib/packaging/hgpackaging/pyoxidizer.py
--- a/contrib/packaging/hgpackaging/pyoxidizer.py
+++ b/contrib/packaging/hgpackaging/pyoxidizer.py
@@ -12,6 +12,7 @@ 
 import shutil
 import subprocess
 import sys
+import typing
 
 from .downloads import download_entry
 from .util import (
@@ -69,7 +70,11 @@ 
 
 
 def run_pyoxidizer(
-    source_dir: pathlib.Path, build_dir: pathlib.Path, target_triple: str,
+    source_dir: pathlib.Path,
+    build_dir: pathlib.Path,
+    target_triple: str,
+    build_vars: typing.Optional[typing.Dict[str, str]] = None,
+    target: typing.Optional[str] = None,
 ) -> pathlib.Path:
     """Run `pyoxidizer` in an environment with access to build dependencies.
 
@@ -77,6 +82,8 @@ 
     artifacts. Actual build artifacts are likely in a sub-directory with the
     name of the pyoxidizer build target that was built.
     """
+    build_vars = build_vars or {}
+
     # We need to make gettext binaries available for compiling i18n files.
     gettext_pkg, gettext_entry = download_entry('gettext', build_dir)
     gettext_dep_pkg = download_entry('gettext-dep', build_dir)[0]
@@ -104,6 +111,12 @@ 
         target_triple,
     ]
 
+    for k, v in sorted(build_vars.items()):
+        args.extend(["--var", k, v])
+
+    if target:
+        args.append(target)
+
     subprocess.run(args, env=env, check=True)
 
     return source_dir / "build" / "pyoxidizer" / target_triple / "release"