Patchwork [5,of,5,stable,osx-packaging,v2] osx: create a modern package including manpages

login
register
mail settings
Submitter Augie Fackler
Date April 27, 2016, 3:45 p.m.
Message ID <265ee42058ff2bc1811f.1461771955@augie-macbookair2.roam.corp.google.com>
Download mbox | patch
Permalink /patch/14805/
State Superseded
Commit db5084d27df5fb19eec75b4b2b57fc3cf499f695
Headers show

Comments

Augie Fackler - April 27, 2016, 3:45 p.m.
# HG changeset patch
# User Kevin Bullock <kbullock+mercurial@ringworld.org>
# Date 1461770436 18000
#      Wed Apr 27 10:20:36 2016 -0500
# Branch stable
# Node ID 265ee42058ff2bc1811f56ca8216d24661c53eeb
# Parent  4834b2dccefc7d1948d9e08035c89e7e30b893d8
osx: create a modern package including manpages

Instead of using bdist_mpkg, we use the modern Apple-provided tools to
build an OS X Installer package directly. This has several advantages:

* Avoids bdist_mpkg which seems to be barely maintained and is hard to
  use.
* Creates a single unified .pkg instead of a .mpkg.
* The package we produce is in the modern, single-file format instead of
  a directory bundle that we have to zip up for download.

In addition, this way of building the package now correctly:

* Installs the manpages, bringing the `make osx`-generated package in
  line with the official Mac packages we publish on the website.
* Installs files with the correct permissions instead of encoding the
  UID of the user who happened to build the package.

Thanks to Augie for updating the test expectations and hghave.
Augie Fackler - April 27, 2016, 3:50 p.m.
Kevin and I got bent out of shape about the OS X packaging and worked out some tests and fixes for the various problems in the existing packages. At this point, I think it's fair to say we'll build the packages for 3.8 with these rules even if they're not committed to stable before 3.8 drops.


> On Apr 27, 2016, at 11:45, Augie Fackler <raf@durin42.com> wrote:
> 
> # HG changeset patch
> # User Kevin Bullock <kbullock+mercurial@ringworld.org>
> # Date 1461770436 18000
> #      Wed Apr 27 10:20:36 2016 -0500
> # Branch stable
> # Node ID 265ee42058ff2bc1811f56ca8216d24661c53eeb
> # Parent  4834b2dccefc7d1948d9e08035c89e7e30b893d8
> osx: create a modern package including manpages
> 
> Instead of using bdist_mpkg, we use the modern Apple-provided tools to
> build an OS X Installer package directly. This has several advantages:
> 
> * Avoids bdist_mpkg which seems to be barely maintained and is hard to
>  use.
> * Creates a single unified .pkg instead of a .mpkg.
> * The package we produce is in the modern, single-file format instead of
>  a directory bundle that we have to zip up for download.
> 
> In addition, this way of building the package now correctly:
> 
> * Installs the manpages, bringing the `make osx`-generated package in
>  line with the official Mac packages we publish on the website.
> * Installs files with the correct permissions instead of encoding the
>  UID of the user who happened to build the package.
> 
> Thanks to Augie for updating the test expectations and hghave.
> 
> diff --git a/Makefile b/Makefile
> --- a/Makefile
> +++ b/Makefile
> @@ -156,15 +156,20 @@ i18n/hg.pot: $(PYFILES) $(DOCFILES) i18n
> # Packaging targets
> 
> osx:
> -	python -c 'import bdist_mpkg.script_bdist_mpkg' || \
> -	   (echo "Missing bdist_mpkg (easy_install bdist_mpkg)"; false)
> -	rm -rf dist/mercurial-*.mpkg
> -	python -m bdist_mpkg.script_bdist_mpkg setup.py --
> -	python contrib/fixpax.py dist/mercurial-*.mpkg/Contents/Packages/*.pkg/Contents/Archive.pax.gz
> -	OUTPUTDIR=$${OUTPUTDIR:=packages/osx}
> -	mkdir -p $$OUTPUTDIR
> -	N=`cd dist && echo mercurial-*.mpkg | sed 's,\.mpkg$$,,'` && hdiutil create -srcfolder dist/$$N.mpkg/ -scrub -volname "$$N" -ov $$OUTPUTDIR/$$N.dmg
> -	[ -n "$$KEEPMPKG" ] && mv dist/mercurial-*.mpkg $$OUTPUTDIR || rm -rf dist/mercurial-*.mpkg
> +	python setup.py install --optimize=1 \
> +	  --root=build/mercurial/ --prefix=/usr/local/ \
> +	  --install-lib=/Library/Python/2.7/site-packages/
> +	make -C doc all install DESTDIR="$(PWD)/build/mercurial/"
> +	mkdir -p $${OUTPUTDIR:-dist}
> +	pkgbuild --root build/mercurial/ --identifier org.mercurial-scm.mercurial \
> +	  build/mercurial.pkg
> +	HGVER=$$((cat build/mercurial/Library/Python/2.7/site-packages/mercurial/__version__.py; echo 'print(version)') | python) && \
> +	OSXVER=$$(sw_vers -productVersion | cut -d. -f1,2) && \
> +	productbuild --distribution contrib/macosx/distribution.xml \
> +	  --package-path build/ \
> +	  --version "$${HGVER}" \
> +	  --resources contrib/macosx/ \
> +	  "$${OUTPUTDIR:-dist/}"/Mercurial-"$${HGVER}"-macosx"$${OSXVER}".pkg
> 
> deb:
> 	contrib/builddeb
> diff --git a/contrib/macosx/distribution.xml b/contrib/macosx/distribution.xml
> new file mode 100644
> --- /dev/null
> +++ b/contrib/macosx/distribution.xml
> @@ -0,0 +1,19 @@
> +<?xml version="1.0" encoding="utf-8" standalone="no"?>
> +<installer-gui-script minSpecVersion="1">
> +  <title>Mercurial SCM</title>
> +  <organization>org.mercurial-scm</organization>
> +  <options customize="never" require-scripts="false" rootVolumeOnly="true" />
> +  <welcome file="Welcome.html" mime-type="text/html" />
> +  <license file="../../COPYING" mime-type="text/plain" />
> +  <readme file="Readme.html" mime-type="text/html" />
> +  <pkg-ref id="org.mercurial-scm.mercurial"
> +           version="0"
> +           auth="root"
> +           onConclusion="none">mercurial.pkg</pkg-ref>
> +  <choices-outline>
> +    <line choice="org.mercurial-scm.mercurial"/>
> +  </choices-outline>
> +  <choice id="org.mercurial-scm.mercurial" visible="false">
> +    <pkg-ref id="org.mercurial-scm.mercurial"/>
> +  </choice>
> +</installer-gui-script>
> diff --git a/tests/hghave.py b/tests/hghave.py
> --- a/tests/hghave.py
> +++ b/tests/hghave.py
> @@ -456,11 +456,15 @@ def has_aix():
> def has_osx():
>     return sys.platform == 'darwin'
> 
> -@check("bdistmpkg", "bdist_mpkg Python Package")
> -def has_bdistmpkg():
> +@check("osxpackaging", "OS X packaging tools")
> +def has_osxpackaging():
>     try:
> -        import bdist_mpkg.script_bdist_mpkg
> -        return True
> +        return (matchoutput('pkgbuild', 'Usage: pkgbuild ', ignorestatus=1)
> +                and matchoutput('productbuild', 'Usage: productbuild ',
> +                                ignorestatus=1)
> +                and matchoutput('lsbom', 'Usage: lsbom', ignorestatus=1)
> +                and matchoutput(
> +                    'xar --help', 'Usage: xar', ignorestatus=1))
>     except ImportError:
>         return False
> 
> diff --git a/tests/test-mac-packages.t b/tests/test-mac-packages.t
> --- a/tests/test-mac-packages.t
> +++ b/tests/test-mac-packages.t
> @@ -1,4 +1,4 @@
> -#require test-repo slow osx bdistmpkg
> +#require test-repo slow osx osxpackaging
>   $ OUTPUTDIR=`pwd`
>   $ export OUTPUTDIR
>   $ KEEPMPKG=yes
> @@ -7,36 +7,37 @@
>   $ cd "$TESTDIR"/..
>   $ make osx > $OUTPUTDIR/build.log 2>&1
>   $ cd $OUTPUTDIR
> -  $ ls -d *.dmg *.mpkg
> -  mercurial-*-macosx10.*.dmg (glob)
> -  mercurial-*-macosx10.*.mpkg (glob)
> +  $ ls -d *.pkg
> +  Mercurial-*-macosx10.*.pkg (glob)
> +
> +  $ xar -xf Mercurial*.pkg
> 
> Gather list of all installed files:
> -  $ find *.mpkg -name Archive.bom | xargs lsbom > boms.txt
> -
> -TODO: update to -f 1,2,3 when we're confident the installed owner of
> -our files is corect. Right now it looks like it's the id of the user
> -that builds the mpkg, which is probably slightly wrong.
> +  $ lsbom mercurial.pkg/Bom > boms.txt
> 
> Spot-check some randomly selected files:
> -  $ grep bdiff boms.txt | cut -d '	' -f 1,2
> -  ./mercurial/bdiff.so	100775
> -  ./mercurial/pure/bdiff.py	100664
> -  ./mercurial/pure/bdiff.pyc	100664
> -  ./mercurial/pure/bdiff.pyo	100664
> -TODO: man pages don't get installed
> -  $ egrep 'man[15]' boms.txt | cut -d '	' -f 1,2
> -  $ grep bser boms.txt | cut -d '	' -f 1,2
> -  ./hgext/fsmonitor/pywatchman/bser.so	100775
> -  ./hgext/fsmonitor/pywatchman/pybser.py	100664
> -  ./hgext/fsmonitor/pywatchman/pybser.pyc	100664
> -  ./hgext/fsmonitor/pywatchman/pybser.pyo	100664
> -  $ grep localrepo boms.txt | cut -d '	' -f 1,2
> -  ./mercurial/localrepo.py	100664
> -  ./mercurial/localrepo.pyc	100664
> -  ./mercurial/localrepo.pyo	100664
> -  $ grep '/hg	' boms.txt | cut -d '	' -f 1,2
> -  ./hg	100775
> +  $ grep bdiff boms.txt | cut -d '	' -f 1,2,3
> +  ./Library/Python/2.7/site-packages/mercurial/bdiff.so	100755	0/0
> +  ./Library/Python/2.7/site-packages/mercurial/pure/bdiff.py	100644	0/0
> +  ./Library/Python/2.7/site-packages/mercurial/pure/bdiff.pyc	100644	0/0
> +  ./Library/Python/2.7/site-packages/mercurial/pure/bdiff.pyo	100644	0/0
> +  $ egrep 'man[15]' boms.txt | cut -d '	' -f 1,2,3
> +  ./usr/local/share/man/man1	40755	0/0
> +  ./usr/local/share/man/man1/hg.1	100644	0/0
> +  ./usr/local/share/man/man5	40755	0/0
> +  ./usr/local/share/man/man5/hgignore.5	100644	0/0
> +  ./usr/local/share/man/man5/hgrc.5	100644	0/0
> +  $ grep bser boms.txt | cut -d '	' -f 1,2,3
> +  ./Library/Python/2.7/site-packages/hgext/fsmonitor/pywatchman/bser.so	100755	0/0
> +  ./Library/Python/2.7/site-packages/hgext/fsmonitor/pywatchman/pybser.py	100644	0/0
> +  ./Library/Python/2.7/site-packages/hgext/fsmonitor/pywatchman/pybser.pyc	100644	0/0
> +  ./Library/Python/2.7/site-packages/hgext/fsmonitor/pywatchman/pybser.pyo	100644	0/0
> +  $ grep localrepo boms.txt | cut -d '	' -f 1,2,3
> +  ./Library/Python/2.7/site-packages/mercurial/localrepo.py	100644	0/0
> +  ./Library/Python/2.7/site-packages/mercurial/localrepo.pyc	100644	0/0
> +  ./Library/Python/2.7/site-packages/mercurial/localrepo.pyo	100644	0/0
> +  $ grep '/hg	' boms.txt | cut -d '	' -f 1,2,3
> +  ./usr/local/bin/hg	100755	0/0
> 
> Note that we're not currently installing any /etc/mercurial stuff,
> including merge-tool configurations.
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
Sean Farley - April 27, 2016, 5:40 p.m.
Augie Fackler <raf@durin42.com> writes:

> # HG changeset patch
> # User Kevin Bullock <kbullock+mercurial@ringworld.org>
> # Date 1461770436 18000
> #      Wed Apr 27 10:20:36 2016 -0500
> # Branch stable
> # Node ID 265ee42058ff2bc1811f56ca8216d24661c53eeb
> # Parent  4834b2dccefc7d1948d9e08035c89e7e30b893d8
> osx: create a modern package including manpages
>
> Instead of using bdist_mpkg, we use the modern Apple-provided tools to
> build an OS X Installer package directly. This has several advantages:
>
> * Avoids bdist_mpkg which seems to be barely maintained and is hard to
>   use.
> * Creates a single unified .pkg instead of a .mpkg.
> * The package we produce is in the modern, single-file format instead of
>   a directory bundle that we have to zip up for download.
>
> In addition, this way of building the package now correctly:
>
> * Installs the manpages, bringing the `make osx`-generated package in
>   line with the official Mac packages we publish on the website.
> * Installs files with the correct permissions instead of encoding the
>   UID of the user who happened to build the package.
>
> Thanks to Augie for updating the test expectations and hghave.

Patches 1-3 look a-ok to me. Patch 4 worries me about the uid. I haven't
tested this patch but it looks ok (and I would like to have packaging in
3.8!). I'll try to to test this patch once I get a virtual machine
setup.
Augie Fackler - April 27, 2016, 8:54 p.m.
> On Apr 27, 2016, at 13:40, Sean Farley <sean@farley.io> wrote:
> 
>> * Installs files with the correct permissions instead of encoding the
>>  UID of the user who happened to build the package.
>> 
>> Thanks to Augie for updating the test expectations and hghave.
> 
> Patches 1-3 look a-ok to me. Patch 4 worries me about the uid. I haven't
> tested this patch but it looks ok (and I would like to have packaging in
> 3.8!). I'll try to to test this patch once I get a virtual machine
> setup.

Sean and I worked this out on irc - he was confused as to the state of things (patch 4 merely documents existing brokenness, which is then fixed in patch 5.)

Patch

diff --git a/Makefile b/Makefile
--- a/Makefile
+++ b/Makefile
@@ -156,15 +156,20 @@  i18n/hg.pot: $(PYFILES) $(DOCFILES) i18n
 # Packaging targets
 
 osx:
-	python -c 'import bdist_mpkg.script_bdist_mpkg' || \
-	   (echo "Missing bdist_mpkg (easy_install bdist_mpkg)"; false)
-	rm -rf dist/mercurial-*.mpkg
-	python -m bdist_mpkg.script_bdist_mpkg setup.py --
-	python contrib/fixpax.py dist/mercurial-*.mpkg/Contents/Packages/*.pkg/Contents/Archive.pax.gz
-	OUTPUTDIR=$${OUTPUTDIR:=packages/osx}
-	mkdir -p $$OUTPUTDIR
-	N=`cd dist && echo mercurial-*.mpkg | sed 's,\.mpkg$$,,'` && hdiutil create -srcfolder dist/$$N.mpkg/ -scrub -volname "$$N" -ov $$OUTPUTDIR/$$N.dmg
-	[ -n "$$KEEPMPKG" ] && mv dist/mercurial-*.mpkg $$OUTPUTDIR || rm -rf dist/mercurial-*.mpkg
+	python setup.py install --optimize=1 \
+	  --root=build/mercurial/ --prefix=/usr/local/ \
+	  --install-lib=/Library/Python/2.7/site-packages/
+	make -C doc all install DESTDIR="$(PWD)/build/mercurial/"
+	mkdir -p $${OUTPUTDIR:-dist}
+	pkgbuild --root build/mercurial/ --identifier org.mercurial-scm.mercurial \
+	  build/mercurial.pkg
+	HGVER=$$((cat build/mercurial/Library/Python/2.7/site-packages/mercurial/__version__.py; echo 'print(version)') | python) && \
+	OSXVER=$$(sw_vers -productVersion | cut -d. -f1,2) && \
+	productbuild --distribution contrib/macosx/distribution.xml \
+	  --package-path build/ \
+	  --version "$${HGVER}" \
+	  --resources contrib/macosx/ \
+	  "$${OUTPUTDIR:-dist/}"/Mercurial-"$${HGVER}"-macosx"$${OSXVER}".pkg
 
 deb:
 	contrib/builddeb
diff --git a/contrib/macosx/distribution.xml b/contrib/macosx/distribution.xml
new file mode 100644
--- /dev/null
+++ b/contrib/macosx/distribution.xml
@@ -0,0 +1,19 @@ 
+<?xml version="1.0" encoding="utf-8" standalone="no"?>
+<installer-gui-script minSpecVersion="1">
+  <title>Mercurial SCM</title>
+  <organization>org.mercurial-scm</organization>
+  <options customize="never" require-scripts="false" rootVolumeOnly="true" />
+  <welcome file="Welcome.html" mime-type="text/html" />
+  <license file="../../COPYING" mime-type="text/plain" />
+  <readme file="Readme.html" mime-type="text/html" />
+  <pkg-ref id="org.mercurial-scm.mercurial"
+           version="0"
+           auth="root"
+           onConclusion="none">mercurial.pkg</pkg-ref>
+  <choices-outline>
+    <line choice="org.mercurial-scm.mercurial"/>
+  </choices-outline>
+  <choice id="org.mercurial-scm.mercurial" visible="false">
+    <pkg-ref id="org.mercurial-scm.mercurial"/>
+  </choice>
+</installer-gui-script>
diff --git a/tests/hghave.py b/tests/hghave.py
--- a/tests/hghave.py
+++ b/tests/hghave.py
@@ -456,11 +456,15 @@  def has_aix():
 def has_osx():
     return sys.platform == 'darwin'
 
-@check("bdistmpkg", "bdist_mpkg Python Package")
-def has_bdistmpkg():
+@check("osxpackaging", "OS X packaging tools")
+def has_osxpackaging():
     try:
-        import bdist_mpkg.script_bdist_mpkg
-        return True
+        return (matchoutput('pkgbuild', 'Usage: pkgbuild ', ignorestatus=1)
+                and matchoutput('productbuild', 'Usage: productbuild ',
+                                ignorestatus=1)
+                and matchoutput('lsbom', 'Usage: lsbom', ignorestatus=1)
+                and matchoutput(
+                    'xar --help', 'Usage: xar', ignorestatus=1))
     except ImportError:
         return False
 
diff --git a/tests/test-mac-packages.t b/tests/test-mac-packages.t
--- a/tests/test-mac-packages.t
+++ b/tests/test-mac-packages.t
@@ -1,4 +1,4 @@ 
-#require test-repo slow osx bdistmpkg
+#require test-repo slow osx osxpackaging
   $ OUTPUTDIR=`pwd`
   $ export OUTPUTDIR
   $ KEEPMPKG=yes
@@ -7,36 +7,37 @@ 
   $ cd "$TESTDIR"/..
   $ make osx > $OUTPUTDIR/build.log 2>&1
   $ cd $OUTPUTDIR
-  $ ls -d *.dmg *.mpkg
-  mercurial-*-macosx10.*.dmg (glob)
-  mercurial-*-macosx10.*.mpkg (glob)
+  $ ls -d *.pkg
+  Mercurial-*-macosx10.*.pkg (glob)
+
+  $ xar -xf Mercurial*.pkg
 
 Gather list of all installed files:
-  $ find *.mpkg -name Archive.bom | xargs lsbom > boms.txt
-
-TODO: update to -f 1,2,3 when we're confident the installed owner of
-our files is corect. Right now it looks like it's the id of the user
-that builds the mpkg, which is probably slightly wrong.
+  $ lsbom mercurial.pkg/Bom > boms.txt
 
 Spot-check some randomly selected files:
-  $ grep bdiff boms.txt | cut -d '	' -f 1,2
-  ./mercurial/bdiff.so	100775
-  ./mercurial/pure/bdiff.py	100664
-  ./mercurial/pure/bdiff.pyc	100664
-  ./mercurial/pure/bdiff.pyo	100664
-TODO: man pages don't get installed
-  $ egrep 'man[15]' boms.txt | cut -d '	' -f 1,2
-  $ grep bser boms.txt | cut -d '	' -f 1,2
-  ./hgext/fsmonitor/pywatchman/bser.so	100775
-  ./hgext/fsmonitor/pywatchman/pybser.py	100664
-  ./hgext/fsmonitor/pywatchman/pybser.pyc	100664
-  ./hgext/fsmonitor/pywatchman/pybser.pyo	100664
-  $ grep localrepo boms.txt | cut -d '	' -f 1,2
-  ./mercurial/localrepo.py	100664
-  ./mercurial/localrepo.pyc	100664
-  ./mercurial/localrepo.pyo	100664
-  $ grep '/hg	' boms.txt | cut -d '	' -f 1,2
-  ./hg	100775
+  $ grep bdiff boms.txt | cut -d '	' -f 1,2,3
+  ./Library/Python/2.7/site-packages/mercurial/bdiff.so	100755	0/0
+  ./Library/Python/2.7/site-packages/mercurial/pure/bdiff.py	100644	0/0
+  ./Library/Python/2.7/site-packages/mercurial/pure/bdiff.pyc	100644	0/0
+  ./Library/Python/2.7/site-packages/mercurial/pure/bdiff.pyo	100644	0/0
+  $ egrep 'man[15]' boms.txt | cut -d '	' -f 1,2,3
+  ./usr/local/share/man/man1	40755	0/0
+  ./usr/local/share/man/man1/hg.1	100644	0/0
+  ./usr/local/share/man/man5	40755	0/0
+  ./usr/local/share/man/man5/hgignore.5	100644	0/0
+  ./usr/local/share/man/man5/hgrc.5	100644	0/0
+  $ grep bser boms.txt | cut -d '	' -f 1,2,3
+  ./Library/Python/2.7/site-packages/hgext/fsmonitor/pywatchman/bser.so	100755	0/0
+  ./Library/Python/2.7/site-packages/hgext/fsmonitor/pywatchman/pybser.py	100644	0/0
+  ./Library/Python/2.7/site-packages/hgext/fsmonitor/pywatchman/pybser.pyc	100644	0/0
+  ./Library/Python/2.7/site-packages/hgext/fsmonitor/pywatchman/pybser.pyo	100644	0/0
+  $ grep localrepo boms.txt | cut -d '	' -f 1,2,3
+  ./Library/Python/2.7/site-packages/mercurial/localrepo.py	100644	0/0
+  ./Library/Python/2.7/site-packages/mercurial/localrepo.pyc	100644	0/0
+  ./Library/Python/2.7/site-packages/mercurial/localrepo.pyo	100644	0/0
+  $ grep '/hg	' boms.txt | cut -d '	' -f 1,2,3
+  ./usr/local/bin/hg	100755	0/0
 
 Note that we're not currently installing any /etc/mercurial stuff,
 including merge-tool configurations.