Patchwork [2,of,3] run-tests: add --rev to run tests with specific version of hg

login
register
mail settings
Submitter timeless
Date May 26, 2016, 2:25 a.m.
Message ID <a66db77a12648b2f7048.1464229550@gcc2-power8.osuosl.org>
Download mbox | patch
Permalink /patch/15209/
State Accepted
Headers show

Comments

timeless - May 26, 2016, 2:25 a.m.
# HG changeset patch
# User timeless <timeless@mozdev.org>
# Date 1460027637 0
#      Thu Apr 07 11:13:57 2016 +0000
# Node ID a66db77a12648b2f70486174be6ead8a0f597532
# Parent  94f71dd6051151a839d8516e7f174757c10a3cf2
# EXP-Topic runtests
# Available At https://bitbucket.org/timeless/mercurial-crew
#              hg pull https://bitbucket.org/timeless/mercurial-crew -r a66db77a1264
run-tests: add --rev to run tests with specific version of hg

This is mostly designed for use outside of hg.

We create a clone in $HGTMP/source containing at least the specified
revision, and applicable tags (in order to ensure that tag information
is available for hg version), using a repository format supported by
the specified version of hg.

Design constraints:
* We can not use "archive" because the version information (used by
setup.py to make mercurial/__version__.py) is lost.
* We have to include the tag commit too, otherwise the version
calculation will mess up (same consumer).
* Old versions of mercurial don't necessarily support the repo format
that the current repo has, so we have to blacklist any unsupported
features.
* Certain versions of mercurial choke on the obsstore format, so we
remove it.


If you want to run tests based on the version of Mercurial you're
targeting, you will probably need a wrapper test, as seen in the
tests provided with this commit.
Augie Fackler - May 29, 2016, 9:04 p.m.
On Thu, May 26, 2016 at 02:25:50AM +0000, timeless wrote:
> # HG changeset patch
> # User timeless <timeless@mozdev.org>
> # Date 1460027637 0
> #      Thu Apr 07 11:13:57 2016 +0000
> # Node ID a66db77a12648b2f70486174be6ead8a0f597532
> # Parent  94f71dd6051151a839d8516e7f174757c10a3cf2
> # EXP-Topic runtests
> # Available At https://bitbucket.org/timeless/mercurial-crew
> #              hg pull https://bitbucket.org/timeless/mercurial-crew -r a66db77a1264
> run-tests: add --rev to run tests with specific version of hg

This is interesting, but I'm a little skeptical of its utility. What
are some sample use cases to justify this rather large feature (in
terms of implementation complexity)?

>
> This is mostly designed for use outside of hg.
>
> We create a clone in $HGTMP/source containing at least the specified
> revision, and applicable tags (in order to ensure that tag information
> is available for hg version), using a repository format supported by
> the specified version of hg.
>
> Design constraints:
> * We can not use "archive" because the version information (used by
> setup.py to make mercurial/__version__.py) is lost.
> * We have to include the tag commit too, otherwise the version
> calculation will mess up (same consumer).
> * Old versions of mercurial don't necessarily support the repo format
> that the current repo has, so we have to blacklist any unsupported
> features.
> * Certain versions of mercurial choke on the obsstore format, so we
> remove it.
>
>
> If you want to run tests based on the version of Mercurial you're
> targeting, you will probably need a wrapper test, as seen in the
> tests provided with this commit.
>
> diff -r 94f71dd60511 -r a66db77a1264 tests/run-tests.py
> --- a/tests/run-tests.py	Wed Mar 09 19:55:45 2016 +0000
> +++ b/tests/run-tests.py	Thu Apr 07 11:13:57 2016 +0000
> @@ -263,6 +263,9 @@
>      parser.add_option("--with-python3", metavar="PYTHON3",
>                        help="Python 3 interpreter (if running under Python 2)"
>                             " (TEMPORARY)")
> +    parser.add_option('--rev', type="string",
> +                      metavar="rev",
> +                      help="run tests using the given revision")
>      parser.add_option('--extra-config-opt', action="append",
>                        help='set the given config opt in the test hgrc')
>      parser.add_option('--random', action="store_true",
> @@ -294,12 +297,16 @@
>
>      if options.with_hg:
>          options.with_hg = canonpath(_bytespath(options.with_hg))
> +        if options.rev:
> +            parser.error('--with-hg and --rev are incompatible')
>          if not (os.path.isfile(options.with_hg) and
>                  os.access(options.with_hg, os.X_OK)):
>              parser.error('--with-hg must specify an executable hg script')
>          if not os.path.basename(options.with_hg) == b'hg':
>              sys.stderr.write('warning: --with-hg should specify an hg script\n')
>      if options.local:
> +        if options.rev:
> +            parser.error('--local and --rev are incompatible')
>          testdir = os.path.dirname(_bytespath(canonpath(sys.argv[0])))
>          hgbin = os.path.join(os.path.dirname(testdir), b'hg')
>          if os.name != 'nt' and not os.access(hgbin, os.X_OK):
> @@ -2335,6 +2342,114 @@
>              script = _bytespath(script)
>              exe = _bytespath(exe)
>          hgroot = os.path.dirname(os.path.dirname(script))
> +        if self.options.rev:
> +            # We need a repository to allow setup to correctly calculate the
> +            # version.
> +            tempsrc = os.path.join(self._hgtmp, b"source")
> +            clone = ['hg', 'clone',
> +                     hgroot,
> +                     tempsrc,
> +                    ]
> +            # We need to find out if each requirement is supported in
> +            # the specified version, otherwise we will have to do a
> +            # format conversion.
> +            # It is supported if a file in mercurial/ has any of:
> +            # (feature) requirements
> +            # requirements (feature)
> +            check = "set:mercurial/*.py and " \
> +                    'grep(r"requirements.*%s|%s.*requirements")'
> +            procs = []
> +            rev = self.options.rev
> +            with open(os.path.join(hgroot, '.hg', 'requires'), 'r') as requires:
> +                for r in requires:
> +                    r = r.strip()
> +                    proc = subprocess.Popen(['hg', 'locate',
> +                                             '--cwd', hgroot,
> +                                             '--rev', rev,
> +                                             check % (r, r),
> +                                            ],
> +                                            stderr=subprocess.STDOUT,
> +                                            stdout=subprocess.PIPE)
> +                    procs.append((r, proc))
> +            badreqs = set()
> +            while procs:
> +                r, proc = procs.pop(0)
> +                ret = proc.wait()
> +                if wifexited(ret):
> +                    ret = os.WEXITSTATUS(ret)
> +                if ret:
> +                    # The requirement wasn't found and will need to be disabled
> +                    badreqs.add(r)
> +            if badreqs:
> +                tag = rev
> +                # If we were given a tag commit, and we just cloned by the
> +                # tag name, we'd get a revision which didn't include the tag.
> +                # And thus setup would not recognize that the revision we were
> +                # using was tagged.
> +                # Look for the tag revision.
> +                proc = subprocess.Popen([
> +                          'hg', 'log',
> +                          '--cwd', hgroot,
> +                          '.hgtags',
> +                          '-r', 'desc("Added tag %s for changeset")' % rev,
> +                          '-T', '{node}',
> +                       ],
> +                       stderr=subprocess.STDOUT,
> +                       stdout=subprocess.PIPE)
> +                ret = proc.wait()
> +                if wifexited(ret):
> +                    ret = os.WEXITSTATUS(ret)
> +                if ret == 0:
> +                    tag, _err = proc.communicate()
> +                # Clone the revision either of the tag if applicable, or
> +                # the plain revision. And update to the requested revision.
> +                # We're pulling to trigger a conversion with features disabled.
> +                clone.extend(['-r', tag,
> +                              '-u', rev,
> +                              '--pull',
> +                             ])
> +                for r in badreqs:
> +                    # There are two common ways to disable a repository format
> +                    # and there isn't a consistent way to determine which
> +                    # applies. Thankfully, unlike obsstore, there's no harm in
> +                    # providing both.
> +                    clone.extend(['--config', 'format.%s=false' % r,
> +                                  '--config', 'format.use%s=false' % r,
> +                                 ])
> +            else:
> +                # The version we've selected supports the repository features,
> +                # so we'll do a full clone (which is faster, and might
> +                # hardlink), and just update to the desired revision:
> +                clone.extend(['-u', self.options.rev,
> +                             ])
> +
> +            proc = subprocess.Popen(clone,
> +                                    stderr=subprocess.STDOUT,
> +                                    stdout=subprocess.PIPE)
> +            ret = proc.wait()
> +            if wifexited(ret):
> +                ret = os.WEXITSTATUS(ret)
> +            if ret:
> +                _out, _err = proc.communicate()
> +                out = b"hg clone failed (%d):\n" % ret
> +                if _out:
> +                    out += _out
> +                if _err:
> +                    out += _err
> +                if PYTHON3:
> +                    sys.stdout.buffer.write(out)
> +                else:
> +                    sys.stdout.write(out)
> +                sys.exit(1)
> +            try:
> +                # The current obbstore version (1) is horribly incompatible
> +                # with certain older versions of hg which are incapable of
> +                # being told to ignore the obsstore. And we don't need an
> +                # obsstore for our purposes.
> +                os.remove(os.path.join(tempsrc, '.hg', 'store', 'obsstore'))
> +            except OSError:
> +                pass
> +            hgroot = tempsrc
>          self._hgroot = hgroot
>          os.chdir(hgroot)
>          nohome = b'--home=""'
> diff -r 94f71dd60511 -r a66db77a1264 tests/test-run-tests-rev.t
> --- /dev/null	Thu Jan 01 00:00:00 1970 +0000
> +++ b/tests/test-run-tests-rev.t	Thu Apr 07 11:13:57 2016 +0000
> @@ -0,0 +1,40 @@
> +#require test-repo slow
> +
> +This file tests the behavior of run-tests.py --rev.
> +
> +  $ . "$TESTDIR/helpers-testrepo.sh"
> +
> +Avoid interference from actual test env:
> +
> +  $ unset HGTEST_JOBS
> +  $ unset HGTEST_TIMEOUT
> +  $ unset HGTEST_PORT
> +  $ unset HGTEST_SHELL
> +
> +support for running a different version of mercurial
> +note that it is typical for ~1/5 tests not to pass when using older
> +versions of mercurial...
> +
> +  $ cat >> test-version.t <<EOF
> +  >   $ hg version
> +  >   Mercurial Distributed SCM (version 1.2)
> +  >
> +  >   Copyright (C) 2005-2008 Matt Mackall <mpm@selenic.com> and others
> +  >   This is free software; see the source for copying conditions. There is NO
> +  >   warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
> +  > EOF
> +  $ run-tests.py --pure --rev 1.2 test-version.t
> +  .
> +  # Ran 1 tests, 0 skipped, 0 warned, 0 failed.
> +
> +To run tests at a version from the revision you select from --rev
> +you need a test which introspects the source directory
> +
> +  $ cat > test-old-basic.t <<EOF
> +  >   $ run-tests.py --pure --local ../../source/tests/test-basic.t
> +  >   .
> +  >   # Ran 1 tests, 0 skipped, 0 warned, 0 failed.
> +  > EOF
> +  $ run-tests.py --pure --rev 1.7 test-old-basic.t
> +  .
> +  # Ran 1 tests, 0 skipped, 0 warned, 0 failed.
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
timeless - May 29, 2016, 10:56 p.m.
On Sun, May 29, 2016 at 5:04 PM, Augie Fackler <raf@durin42.com> wrote:
> This is interesting, but I'm a little skeptical of its utility. What
> are some sample use cases to justify this rather large feature (in
> terms of implementation complexity)?

I was thinking of using this for evolve testing (which tests against
multiple hg versions) and for hg book annotations (determining when
features were added).

Some of the code probably belongs in hg itself -- It shouldn't be this
complicated to create a clone repo supported by an identifiable
version of hg.

Patch

diff -r 94f71dd60511 -r a66db77a1264 tests/run-tests.py
--- a/tests/run-tests.py	Wed Mar 09 19:55:45 2016 +0000
+++ b/tests/run-tests.py	Thu Apr 07 11:13:57 2016 +0000
@@ -263,6 +263,9 @@ 
     parser.add_option("--with-python3", metavar="PYTHON3",
                       help="Python 3 interpreter (if running under Python 2)"
                            " (TEMPORARY)")
+    parser.add_option('--rev', type="string",
+                      metavar="rev",
+                      help="run tests using the given revision")
     parser.add_option('--extra-config-opt', action="append",
                       help='set the given config opt in the test hgrc')
     parser.add_option('--random', action="store_true",
@@ -294,12 +297,16 @@ 
 
     if options.with_hg:
         options.with_hg = canonpath(_bytespath(options.with_hg))
+        if options.rev:
+            parser.error('--with-hg and --rev are incompatible')
         if not (os.path.isfile(options.with_hg) and
                 os.access(options.with_hg, os.X_OK)):
             parser.error('--with-hg must specify an executable hg script')
         if not os.path.basename(options.with_hg) == b'hg':
             sys.stderr.write('warning: --with-hg should specify an hg script\n')
     if options.local:
+        if options.rev:
+            parser.error('--local and --rev are incompatible')
         testdir = os.path.dirname(_bytespath(canonpath(sys.argv[0])))
         hgbin = os.path.join(os.path.dirname(testdir), b'hg')
         if os.name != 'nt' and not os.access(hgbin, os.X_OK):
@@ -2335,6 +2342,114 @@ 
             script = _bytespath(script)
             exe = _bytespath(exe)
         hgroot = os.path.dirname(os.path.dirname(script))
+        if self.options.rev:
+            # We need a repository to allow setup to correctly calculate the
+            # version.
+            tempsrc = os.path.join(self._hgtmp, b"source")
+            clone = ['hg', 'clone',
+                     hgroot,
+                     tempsrc,
+                    ]
+            # We need to find out if each requirement is supported in
+            # the specified version, otherwise we will have to do a
+            # format conversion.
+            # It is supported if a file in mercurial/ has any of:
+            # (feature) requirements
+            # requirements (feature)
+            check = "set:mercurial/*.py and " \
+                    'grep(r"requirements.*%s|%s.*requirements")'
+            procs = []
+            rev = self.options.rev
+            with open(os.path.join(hgroot, '.hg', 'requires'), 'r') as requires:
+                for r in requires:
+                    r = r.strip()
+                    proc = subprocess.Popen(['hg', 'locate',
+                                             '--cwd', hgroot,
+                                             '--rev', rev,
+                                             check % (r, r),
+                                            ],
+                                            stderr=subprocess.STDOUT,
+                                            stdout=subprocess.PIPE)
+                    procs.append((r, proc))
+            badreqs = set()
+            while procs:
+                r, proc = procs.pop(0)
+                ret = proc.wait()
+                if wifexited(ret):
+                    ret = os.WEXITSTATUS(ret)
+                if ret:
+                    # The requirement wasn't found and will need to be disabled
+                    badreqs.add(r)
+            if badreqs:
+                tag = rev
+                # If we were given a tag commit, and we just cloned by the
+                # tag name, we'd get a revision which didn't include the tag.
+                # And thus setup would not recognize that the revision we were
+                # using was tagged.
+                # Look for the tag revision.
+                proc = subprocess.Popen([
+                          'hg', 'log',
+                          '--cwd', hgroot,
+                          '.hgtags',
+                          '-r', 'desc("Added tag %s for changeset")' % rev,
+                          '-T', '{node}',
+                       ],
+                       stderr=subprocess.STDOUT,
+                       stdout=subprocess.PIPE)
+                ret = proc.wait()
+                if wifexited(ret):
+                    ret = os.WEXITSTATUS(ret)
+                if ret == 0:
+                    tag, _err = proc.communicate()
+                # Clone the revision either of the tag if applicable, or
+                # the plain revision. And update to the requested revision.
+                # We're pulling to trigger a conversion with features disabled.
+                clone.extend(['-r', tag,
+                              '-u', rev,
+                              '--pull',
+                             ])
+                for r in badreqs:
+                    # There are two common ways to disable a repository format
+                    # and there isn't a consistent way to determine which
+                    # applies. Thankfully, unlike obsstore, there's no harm in
+                    # providing both.
+                    clone.extend(['--config', 'format.%s=false' % r,
+                                  '--config', 'format.use%s=false' % r,
+                                 ])
+            else:
+                # The version we've selected supports the repository features,
+                # so we'll do a full clone (which is faster, and might
+                # hardlink), and just update to the desired revision:
+                clone.extend(['-u', self.options.rev,
+                             ])
+
+            proc = subprocess.Popen(clone,
+                                    stderr=subprocess.STDOUT,
+                                    stdout=subprocess.PIPE)
+            ret = proc.wait()
+            if wifexited(ret):
+                ret = os.WEXITSTATUS(ret)
+            if ret:
+                _out, _err = proc.communicate()
+                out = b"hg clone failed (%d):\n" % ret
+                if _out:
+                    out += _out
+                if _err:
+                    out += _err
+                if PYTHON3:
+                    sys.stdout.buffer.write(out)
+                else:
+                    sys.stdout.write(out)
+                sys.exit(1)
+            try:
+                # The current obbstore version (1) is horribly incompatible
+                # with certain older versions of hg which are incapable of
+                # being told to ignore the obsstore. And we don't need an
+                # obsstore for our purposes.
+                os.remove(os.path.join(tempsrc, '.hg', 'store', 'obsstore'))
+            except OSError:
+                pass
+            hgroot = tempsrc
         self._hgroot = hgroot
         os.chdir(hgroot)
         nohome = b'--home=""'
diff -r 94f71dd60511 -r a66db77a1264 tests/test-run-tests-rev.t
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/tests/test-run-tests-rev.t	Thu Apr 07 11:13:57 2016 +0000
@@ -0,0 +1,40 @@ 
+#require test-repo slow
+
+This file tests the behavior of run-tests.py --rev.
+
+  $ . "$TESTDIR/helpers-testrepo.sh"
+
+Avoid interference from actual test env:
+
+  $ unset HGTEST_JOBS
+  $ unset HGTEST_TIMEOUT
+  $ unset HGTEST_PORT
+  $ unset HGTEST_SHELL
+
+support for running a different version of mercurial
+note that it is typical for ~1/5 tests not to pass when using older
+versions of mercurial...
+
+  $ cat >> test-version.t <<EOF
+  >   $ hg version
+  >   Mercurial Distributed SCM (version 1.2)
+  >   
+  >   Copyright (C) 2005-2008 Matt Mackall <mpm@selenic.com> and others
+  >   This is free software; see the source for copying conditions. There is NO
+  >   warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
+  > EOF
+  $ run-tests.py --pure --rev 1.2 test-version.t
+  .
+  # Ran 1 tests, 0 skipped, 0 warned, 0 failed.
+
+To run tests at a version from the revision you select from --rev
+you need a test which introspects the source directory
+
+  $ cat > test-old-basic.t <<EOF
+  >   $ run-tests.py --pure --local ../../source/tests/test-basic.t
+  >   .
+  >   # Ran 1 tests, 0 skipped, 0 warned, 0 failed.
+  > EOF
+  $ run-tests.py --pure --rev 1.7 test-old-basic.t
+  .
+  # Ran 1 tests, 0 skipped, 0 warned, 0 failed.