Patchwork [9,of,9,debian-packaging] debian: switch to using debhelper and dh_python2 to build debs

login
register
mail settings
Submitter Augie Fackler
Date Aug. 26, 2015, 7:08 p.m.
Message ID <123c754a4606db14891f.1440616130@arthedain.pit.corp.google.com>
Download mbox | patch
Permalink /patch/10292/
State Superseded
Commit 7f49efcaa9b4d870ef037401cfded0135701392f
Headers show

Comments

Augie Fackler - Aug. 26, 2015, 7:08 p.m.
# HG changeset patch
# User Augie Fackler <augie@google.com>
# Date 1440601149 14400
#      Wed Aug 26 10:59:09 2015 -0400
# Node ID 123c754a4606db14891ff9653e838c15a9b08bbd
# Parent  16a96256683d535ad64a628d30849ce7a326cf70
debian: switch to using debhelper and dh_python2 to build debs

This is a much larger commit than I'd like, but I honestly don't see a
good way to break it up and leave things working. Summary:

We now use debian/rules with debhelper to build our debs. This is much
more standard, and means we use dh_python2 to do things like handle
leaving .pyc files out of the built debs.

The resulting package is split into mercurial and mercurial-common,
with the former being the hg stub and all the native .sos, and the
latter being basically everything else.

builddeb and dockerdeb are updated to use the new system. The old way
(using dpkg by hand) breaks with the above changes because
debian/control no longer contains a version string (that's now guessed
from the phony changelog.)

Tests are updated to assert that the right files end up in the right
debs.
Matt Mackall - Aug. 26, 2015, 11:49 p.m.
On Wed, 2015-08-26 at 15:08 -0400, Augie Fackler wrote:
> # HG changeset patch
> # User Augie Fackler <augie@google.com>
> # Date 1440601149 14400
> #      Wed Aug 26 10:59:09 2015 -0400
> # Node ID 123c754a4606db14891ff9653e838c15a9b08bbd
> # Parent  16a96256683d535ad64a628d30849ce7a326cf70
> debian: switch to using debhelper and dh_python2 to build debs

Having a bit of trouble with this set. First:

tests/test-debian-packages.t:4:
 >   $ export DEB_BUILD_OPTIONS=nocheck
 don't export and assign at once
*** check-code complained ***

This test doesn't actually run here (debian-unstable), because while I
do obviously have dpkg, I don't have debuild and friends. Also, since
this is building for jessie, it's eventually going to stop working on
>jessie.
Augie Fackler - Aug. 26, 2015, 11:56 p.m.
> On Aug 26, 2015, at 7:49 PM, Matt Mackall <mpm@selenic.com> wrote:
> 
> On Wed, 2015-08-26 at 15:08 -0400, Augie Fackler wrote:
>> # HG changeset patch
>> # User Augie Fackler <augie@google.com>
>> # Date 1440601149 14400
>> #      Wed Aug 26 10:59:09 2015 -0400
>> # Node ID 123c754a4606db14891ff9653e838c15a9b08bbd
>> # Parent  16a96256683d535ad64a628d30849ce7a326cf70
>> debian: switch to using debhelper and dh_python2 to build debs
> 
> Having a bit of trouble with this set. First:
> 
> tests/test-debian-packages.t:4:
>>  $ export DEB_BUILD_OPTIONS=nocheck
> don't export and assign at once
> *** check-code complained ***

Bah. Forgot to run whole test suite after iterating on the packaging test.

> 
> This test doesn't actually run here (debian-unstable), because while I
> do obviously have dpkg, I don't have debuild and friends.

Yup, I’ll add some hghave checks and resend. I assume you took none of the series?

> Also, since
> this is building for jessie, it's eventually going to stop working on
> jessie.

Yeah, I think I want to rename the build rules. I think we should have ‘make deb’ which builds a deb for the current platform, and then ‘make docker-debian-jessie' that builds specifically for jessie. Sound good?

> 
> --
> Mathematics is the supreme nostalgia of our time.
>
Matt Mackall - Aug. 27, 2015, 12:03 a.m.
On Wed, 2015-08-26 at 19:56 -0400, Augie Fackler wrote:
> > On Aug 26, 2015, at 7:49 PM, Matt Mackall <mpm@selenic.com> wrote:
> > 
> > On Wed, 2015-08-26 at 15:08 -0400, Augie Fackler wrote:
> >> # HG changeset patch
> >> # User Augie Fackler <augie@google.com>
> >> # Date 1440601149 14400
> >> #      Wed Aug 26 10:59:09 2015 -0400
> >> # Node ID 123c754a4606db14891ff9653e838c15a9b08bbd
> >> # Parent  16a96256683d535ad64a628d30849ce7a326cf70
> >> debian: switch to using debhelper and dh_python2 to build debs
> > 
> > Having a bit of trouble with this set. First:
> > 
> > tests/test-debian-packages.t:4:
> >>  $ export DEB_BUILD_OPTIONS=nocheck
> > don't export and assign at once
> > *** check-code complained ***
> 
> Bah. Forgot to run whole test suite after iterating on the packaging test.
> 
> > 
> > This test doesn't actually run here (debian-unstable), because while I
> > do obviously have dpkg, I don't have debuild and friends.
> 
> Yup, I’ll add some hghave checks and resend. I assume you took none of the series?

Not yet. 

> > Also, since
> > this is building for jessie, it's eventually going to stop working on
> > jessie.
> 
> Yeah, I think I want to rename the build rules. I think we should have
> ‘make deb’ which builds a deb for the current platform, and then ‘make
> docker-debian-jessie' that builds specifically for jessie. Sound good?

Sounds fine, but I don't know if we'll run into Debian vs Ubuntu/whatnot
issues?
Augie Fackler - Aug. 27, 2015, 12:05 a.m.
> On Aug 26, 2015, at 8:03 PM, Matt Mackall <mpm@selenic.com> wrote:
> 
> On Wed, 2015-08-26 at 19:56 -0400, Augie Fackler wrote:
>>> On Aug 26, 2015, at 7:49 PM, Matt Mackall <mpm@selenic.com> wrote:
>>> 
>>> On Wed, 2015-08-26 at 15:08 -0400, Augie Fackler wrote:
>>>> # HG changeset patch
>>>> # User Augie Fackler <augie@google.com>
>>>> # Date 1440601149 14400
>>>> #      Wed Aug 26 10:59:09 2015 -0400
>>>> # Node ID 123c754a4606db14891ff9653e838c15a9b08bbd
>>>> # Parent  16a96256683d535ad64a628d30849ce7a326cf70
>>>> debian: switch to using debhelper and dh_python2 to build debs
>>> 
>>> Having a bit of trouble with this set. First:
>>> 
>>> tests/test-debian-packages.t:4:
>>>> $ export DEB_BUILD_OPTIONS=nocheck
>>> don't export and assign at once
>>> *** check-code complained ***
>> 
>> Bah. Forgot to run whole test suite after iterating on the packaging test.
>> 
>>> 
>>> This test doesn't actually run here (debian-unstable), because while I
>>> do obviously have dpkg, I don't have debuild and friends.
>> 
>> Yup, I’ll add some hghave checks and resend. I assume you took none of the series?
> 
> Not yet.
> 
>>> Also, since
>>> this is building for jessie, it's eventually going to stop working on
>>> jessie.
>> 
>> Yeah, I think I want to rename the build rules. I think we should have
>> ‘make deb’ which builds a deb for the current platform, and then ‘make
>> docker-debian-jessie' that builds specifically for jessie. Sound good?
> 
> Sounds fine, but I don't know if we'll run into Debian vs Ubuntu/whatnot
> issues?

I honestly have no idea. My packages that I built using docker for jessie worked fine on ubuntu LTS, and the Mercurial there is ancient, so I *suspect* we’ll be mostly okay.

I’ll do a resend of the whole series with some extra tweaking (rename non-docker target, fix up hghave, etc)

> 
> --
> Mathematics is the supreme nostalgia of our time.
>

Patch

diff --git a/contrib/builddeb b/contrib/builddeb
--- a/contrib/builddeb
+++ b/contrib/builddeb
@@ -7,6 +7,7 @@ 
 . $(dirname $0)/packagelib.sh
 
 BUILD=1
+CLEANUP=1
 DEBVERSION=jessie
 while [ "$1" ]; do
     case "$1" in
@@ -15,10 +16,14 @@  while [ "$1" ]; do
         DEBVERSION="$1"
         shift
         ;;
-    --prepare )
+    --cleanup )
         shift
         BUILD=
         ;;
+    --build )
+        shift
+        CLEANUP=
+        ;;
     --debbuilddir )
         shift
         DEBBUILDDIR="$1"
@@ -31,13 +36,10 @@  while [ "$1" ]; do
     esac
 done
 
-DEBBUILDDIR=${OUTPUTDIR:="$PWD/debbuild"}
+trap "if [ '$CLEANUP' ] ; then rm -r '$PWD/debian' ; fi" EXIT
 
 set -u
 
-rm -rf $DEBBUILDDIR
-mkdir -p $DEBBUILDDIR
-
 if [ ! -d .hg ]; then
     echo 'You are not inside a Mercurial repository!' 1>&2
     exit 1
@@ -45,26 +47,38 @@  fi
 
 gethgversion
 
-cp -r $PWD/contrib/debian $DEBBUILDDIR/DEBIAN
-chmod -R 0755 $DEBBUILDDIR/DEBIAN
-
-control=$DEBBUILDDIR/DEBIAN/control
-
-# This looks like sed -i, but sed -i behaves just differently enough
-# between BSD and GNU sed that I gave up and did the dumb thing.
-sed "s/__VERSION__/$version/" < $control > $control.tmp
-mv $control.tmp $control
+control=debian/control
+changelog=debian/changelog
 
 if [ "$BUILD" ]; then
-    make PREFIX=$DEBBUILDDIR/usr install
-    dpkg-deb --build $DEBBUILDDIR
-    mv $DEBBUILDDIR.deb $DEBBUILDDIR/mercurial-$version-$release.deb
-    if [ $? = 0 ]; then
-        echo
-        echo "Built packages for $version-$release:"
-        find $DEBBUILDDIR/ -type f -newer $control
+    if [ -d debian ] ; then
+        echo "Error! debian control directory already exists!"
+        exit 1
     fi
-else
-    echo "Prepared sources for $version-$release $control are in $DEBBUILDDIR - use like:"
-    echo "dpkg-deb --build $DEBBUILDDIR"
+
+    cp -r $PWD/contrib/debian debian
+    chmod -R 0755 debian
+
+    # This looks like sed -i, but sed -i behaves just differently enough
+    # between BSD and GNU sed that I gave up and did the dumb thing.
+    sed "s/__VERSION__/$version/" < $changelog > $changelog.tmp
+    date=$(date --rfc-2822)
+    sed "s/__DATE__/$date/" < $changelog.tmp > $changelog
+    rm $changelog.tmp
+
+    debuild -us -uc -b
+    if [ $? != 0 ]; then
+        echo 'debuild failed!'
+        exit 1
+    fi
+
 fi
+if [ "$CLEANUP" ] ; then
+    echo
+    OUTPUTDIR=${OUTPUTDIR:=packages/debian-$DEBVERSION}
+    find ../mercurial*.deb ../mercurial_*.build ../mercurial_*.changes \
+          -type f -newer $control -print0 | \
+      xargs -Inarf -0 mv narf "$OUTPUTDIR"
+    echo "Built packages for $version-$release:"
+    find "$OUTPUTDIR" -type f -newer $control -name '*.deb'
+fi
diff --git a/contrib/debian/changelog b/contrib/debian/changelog
new file mode 100644
--- /dev/null
+++ b/contrib/debian/changelog
@@ -0,0 +1,5 @@ 
+mercurial (__VERSION__) unstable; urgency=medium
+
+  * Automated build performed by upstream.
+
+ -- Mercurial Devel <mercurial-devel@selenic.com>  __DATE__
diff --git a/contrib/debian/compat b/contrib/debian/compat
new file mode 100644
--- /dev/null
+++ b/contrib/debian/compat
@@ -0,0 +1,1 @@ 
+9
diff --git a/contrib/debian/control b/contrib/debian/control
--- a/contrib/debian/control
+++ b/contrib/debian/control
@@ -1,9 +1,47 @@ 
-Package: mercurial
-Version: __VERSION__
+Source: mercurial
 Section: vcs
 Priority: optional
+Maintainer: Mercurial Developers <mercurial-devel@selenic.com>
+Build-Depends:
+ debhelper (>= 7),
+ dh-python,
+ python-all
+Standards-Version: 3.9.4
+X-Python-Version: >= 2.6
+
+Package: mercurial
+Depends:
+ python,
+ ${shlibs:Depends},
+ ${misc:Depends},
+ ${python:Depends},
+ mercurial-common (= ${source:Version})
+Architecture: any
+Description: fast, easy to use, distributed revision control tool.
+ Mercurial is a fast, lightweight Source Control Management system designed
+ for efficient handling of very large distributed projects.
+ .
+ Its features include:
+  * O(1) delta-compressed file storage and retrieval scheme
+  * Complete cross-indexing of files and changesets for efficient exploration
+    of project history
+  * Robust SHA1-based integrity checking and append-only storage model
+  * Decentralized development model with arbitrary merging between trees
+  * Easy-to-use command-line interface
+  * Integrated stand-alone web interface
+  * Small Python codebase
+
+Package: mercurial-common
 Architecture: all
-Depends: python
-Conflicts: mercurial-common
-Maintainer: Mercurial Developers <mercurial-devel@selenic.com>
-Description: Mercurial (probably nightly) package built by upstream.
+Depends:
+ ${misc:Depends},
+ ${python:Depends},
+Recommends: mercurial (= ${source:Version}), ca-certificates
+Breaks: mercurial (<< ${source:Version})
+Replaces: mercurial (<< 2.6.3)
+Description: easy-to-use, scalable distributed version control system (common files)
+ Mercurial is a fast, lightweight Source Control Management system designed
+ for efficient handling of very large distributed projects.
+ .
+ This package contains the architecture independent components of Mercurial,
+ and is generally useless without the mercurial package.
diff --git a/contrib/debian/copyright b/contrib/debian/copyright
new file mode 100644
--- /dev/null
+++ b/contrib/debian/copyright
@@ -0,0 +1,27 @@ 
+Format: http://www.debian.org/doc/packaging-manuals/copyright-format/1.0/
+Upstream-Name: mercurial
+Source: http://www.selenic.com/mercurial/
+
+Files: *
+Copyright: 2005-2015, Matt Mackall <mpm@selenic.com> and others.
+License: GPL-2+
+ This program is free software; you can redistribute it
+ and/or modify it under the terms of the GNU General Public
+ License as published by the Free Software Foundation; either
+ version 2 of the License, or (at your option) any later
+ version.
+ .
+ This program is distributed in the hope that it will be
+ useful, but WITHOUT ANY WARRANTY; without even the implied
+ warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR
+ PURPOSE.  See the GNU General Public License for more
+ details.
+ .
+ You should have received a copy of the GNU General Public
+ License along with this package; if not, write to the Free
+ Software Foundation, Inc., 51 Franklin St, Fifth Floor,
+ Boston, MA  02110-1301 USA
+ .
+ On Debian systems, the full text of the GNU General Public
+ License version 2 can be found in the file
+ `/usr/share/common-licenses/GPL-2'.
diff --git a/contrib/debian/rules b/contrib/debian/rules
new file mode 100644
--- /dev/null
+++ b/contrib/debian/rules
@@ -0,0 +1,29 @@ 
+#!/usr/bin/make -f
+# Uncomment this to turn on verbose mode.
+# export DH_VERBOSE=1
+
+CPUS=$(shell cat /proc/cpuinfo | grep -E ^processor | wc -l)
+
+%:
+	dh $@ --with python2
+
+override_dh_auto_test:
+	http_proxy='' dh_auto_test -- TESTFLAGS="-j$(CPUS)"
+
+override_dh_python2:
+	dh_python2
+	find debian/mercurial/usr/share -type d -empty -delete
+
+override_dh_install:
+	python$(PYVERS) setup.py install --root $(CURDIR)/debian/mercurial --install-layout=deb
+	# remove arch-independent python stuff
+	find $(CURDIR)/debian/mercurial/usr/lib \
+		! -name '*.so' ! -type d -delete , \
+		-type d -empty -delete
+	python$(PYVERS) setup.py install --root $(CURDIR)/debian/mercurial-common --install-layout=deb
+	make install-doc PREFIX=$(CURDIR)/debian/mercurial-common/usr
+	# remove arch-dependent python stuff
+	find $(CURDIR)/debian/mercurial-common/usr/lib \
+		-name '*.so' ! -type d -delete , \
+		-type d -empty -delete
+	rm $(CURDIR)/debian/mercurial-common/usr/bin/hg
diff --git a/contrib/dockerdeb b/contrib/dockerdeb
--- a/contrib/dockerdeb
+++ b/contrib/dockerdeb
@@ -12,29 +12,23 @@  DEBPLATFORM="$1"
 PLATFORM="debian-$1"
 shift # extra params are passed to build process
 
+OUTPUTDIR=${OUTPUTDIR:=$ROOTDIR/packages/$PLATFORM}
+
 initcontainer $PLATFORM
 
-DEBBUILDDIR=${OUTPUTDIR:=$ROOTDIR/packages/$PLATFORM}
-OUTPUTDIR=$DEBBUILDDIR/staged contrib/builddeb --release $DEBPLATFORM --prepare
+# debuild only appears to be able to save built debs etc to .., so we
+# have to share the .. of the current directory with the docker
+# container and hope it's writable. Whee.
+dn=$(basename $PWD)
 
-DSHARED=/mnt/shared/
 if [ $(uname) = "Darwin" ] ; then
-    $DOCKER run -u $DBUILDUSER --rm -v $DEBBUILDDIR:$DSHARED -v $PWD:/mnt/hg $CONTAINER \
-            sh -c "cd /mnt/hg && make clean && make local"
+    $DOCKER run -u $DBUILDUSER --rm -v $PWD/..:/mnt $CONTAINER \
+            sh -c "cd /mnt/$dn && make clean && make local"
 fi
-$DOCKER run -u $DBUILDUSER --rm -v $DEBBUILDDIR:$DSHARED -v $PWD:/mnt/hg $CONTAINER \
-  sh -c "cd /mnt/hg && make PREFIX=$DSHARED/staged/usr install"
-$DOCKER run -u $DBUILDUSER --rm -v $DEBBUILDDIR:$DSHARED $CONTAINER \
-  dpkg-deb --build $DSHARED/staged
+$DOCKER run -u $DBUILDUSER --rm -v $PWD/..:/mnt $CONTAINER \
+  sh -c "cd /mnt/$dn && DEB_BUILD_OPTIONS='$DEB_BUILD_OPTIONS' contrib/builddeb --build --release $DEBPLATFORM"
+contrib/builddeb --cleanup --release $DEBPLATFORM
 if [ $(uname) = "Darwin" ] ; then
-    $DOCKER run -u $DBUILDUSER --rm -v $DEBBUILDDIR:$DSHARED -v $PWD:/mnt/hg $CONTAINER \
-            sh -c "cd /mnt/hg && make clean"
+    $DOCKER run -u $DBUILDUSER --rm -v $PWD/..:/mnt $CONTAINER \
+            sh -c "cd /mnt/$dn && make clean"
 fi
-
-gethgversion
-
-rm -r $DEBBUILDDIR/staged
-mv $DEBBUILDDIR/staged.deb $DEBBUILDDIR/mercurial-$version-$release.deb
-
-echo
-echo "Build complete - results can be found in $DEBBUILDDIR"
diff --git a/tests/test-debian-packages.t b/tests/test-debian-packages.t
--- a/tests/test-debian-packages.t
+++ b/tests/test-debian-packages.t
@@ -1,12 +1,21 @@ 
 #require test-repo slow
 
+Ensure debuild doesn't run the testsuite, as that could get silly.
+  $ export DEB_BUILD_OPTIONS=nocheck
+
 We check debian package contents with portable tools so that when
 we're on non-debian machines we can still test the packages that are
 built using docker.
   $ checkdeb() {
   >   ls *.deb
-  >   ar x mercurial*.deb
-  >   tar tf data.tar* | grep localrepo | sort
+  >   echo main deb should have .so but no .py
+  >   ar x mercurial_*.deb
+  >   tar tf data.tar* | grep parsers
+  >   tar tf data.tar* | grep localrepo
+  >   echo common deb should have .py but no .so
+  >   ar x mercurial-common_*.deb
+  >   tar tf data.tar* | grep parsers
+  >   tar tf data.tar* | grep localrepo
   > }
   $ OUTPUTDIR=`pwd`
   $ export OUTPUTDIR
@@ -16,9 +25,12 @@  built using docker.
   $ make docker-debian-jessie > $OUTPUTDIR/build.log 2>&1
   $ cd $OUTPUTDIR
   $ checkdeb
-  mercurial-*.deb (glob)
-  ./usr/lib/python2.7/site-packages/mercurial/localrepo.py
-  ./usr/lib/python2.7/site-packages/mercurial/localrepo.pyc
+  mercurial-common_*.deb (glob)
+  mercurial_*.deb (glob)
+  main deb should have .so but no .py
+  ./usr/lib/python2.7/dist-packages/mercurial/parsers*.so (glob)
+  common deb should have .py but no .so
+  ./usr/lib/python2.7/dist-packages/mercurial/localrepo.py
   $ rm -f *.deb build.log
 #endif
 
@@ -27,8 +39,11 @@  built using docker.
   $ make debian-jessie > $OUTPUTDIR/build.log 2>&1
   $ cd $OUTPUTDIR
   $ checkdeb
-  mercurial-*.deb (glob)
-  ./usr/lib/python2.7/site-packages/mercurial/localrepo.py
-  ./usr/lib/python2.7/site-packages/mercurial/localrepo.pyc
+  mercurial-common_*.deb (glob)
+  mercurial_*.deb (glob)
+  main deb should have .so but no .py
+  ./usr/lib/python2.7/dist-packages/mercurial/parsers*.so (glob)
+  common deb should have .py but no .so
+  ./usr/lib/python2.7/dist-packages/mercurial/localrepo.py
   $ rm -f *.deb build.log
 #endif