Patchwork extensions: add onsub extension

login
register
mail settings
Submitter Angel Ezquerra
Date Sept. 12, 2014, 11:39 p.m.
Message ID <507d196c9b1c823edbb6.1410565195@Angels-MacBook-Pro.local>
Download mbox | patch
Permalink /patch/5814/
State Changes Requested
Headers show

Comments

Angel Ezquerra - Sept. 12, 2014, 11:39 p.m.
# HG changeset patch
# User Angel Ezquerra <angel.ezquerra@gmail.com>
# Date 1409641307 -7200
#      Tue Sep 02 09:01:47 2014 +0200
# Node ID 507d196c9b1c823edbb6bf8f220acd0b0d734100
# Parent  ca854cd4a26a8770fbc22b4d7ee1ac6823b682a5
extensions: add onsub extension

The onsub extension lets you execute a command in each subrepository. This is
useful when working with a repository that contains a large number of
subrepositories. For example, it can be used for quickly updating all
subrepositories:

  $ hg onsub 'hg pull --update'

This extension was originally written by Martin Geisler for aragost Trifork. The
original repository is on bitbucket (https://bitbucket.org/lantiq/onsub/).
The extension is quite simple (it has not required any changes since 2012, and
it continues working fine to this day). The extension has a couple of tests.

I discussed the inclusion of this extension with Martin Geisler, the original
author of the extension, during the Munich 3.2 sprint. He agreed to the
inclusion of the extension, which I will try to maintain from now on.

Also note that this extension has been included with TortoiseHg for several
major releases.

The code for the extension that included on this patch is identical to the
latest version of the extension on its bitbucket repository. I had to make a
few small changes to the tests:

1. some changes were needed because now the extension is in the hgext folder,
2. adapt to a small change to the --verbose hint on the extension help
3. adapt to a change on the output of mercurial's commit --subrepos command

None of these changes is related to the functionality of the extension itself,
which as I said has not changed in a couple of years.
Angel Ezquerra - Oct. 12, 2014, 2:07 p.m.
On Wed, Sep 17, 2014 at 6:03 AM, Kevin Bullock
<kbullock+mercurial@ringworld.org> wrote:
> On Sep 12, 2014, at 6:39 PM, Angel Ezquerra <angel.ezquerra@gmail.com> wrote:
>
>> # HG changeset patch
>> # User Angel Ezquerra <angel.ezquerra@gmail.com>
>> # Date 1409641307 -7200
>> #      Tue Sep 02 09:01:47 2014 +0200
>> # Node ID 507d196c9b1c823edbb6bf8f220acd0b0d734100
>> # Parent  ca854cd4a26a8770fbc22b4d7ee1ac6823b682a5
>> extensions: add onsub extension
>>
>> The onsub extension lets you execute a command in each subrepository. This is
>> useful when working with a repository that contains a large number of
>> subrepositories. For example, it can be used for quickly updating all
>> subrepositories:
>>
>>  $ hg onsub 'hg pull --update'
>>
>> This extension was originally written by Martin Geisler for aragost Trifork. The
>> original repository is on bitbucket (https://bitbucket.org/lantiq/onsub/).
>> The extension is quite simple (it has not required any changes since 2012, and
>> it continues working fine to this day). The extension has a couple of tests.
>>
>> I discussed the inclusion of this extension with Martin Geisler, the original
>> author of the extension, during the Munich 3.2 sprint. He agreed to the
>> inclusion of the extension, which I will try to maintain from now on.
>>
>> Also note that this extension has been included with TortoiseHg for several
>> major releases.
>>
>> The code for the extension that included on this patch is identical to the
>> latest version of the extension on its bitbucket repository. I had to make a
>> few small changes to the tests:
>
> I'm generally in favor of bringing this in if you'll take on maintenance. We have more accurate, history-proof ways of spelling "latest version" though. Please reference a specific changeset in the source repo.
>
> Haven't much looked at the code itself yet.

Kevin,

I thought I had replied to this email, but it seems I did not. I will
resend the patch with a change to the commit message indicating the
short hash of the revision on the original onsub extension repository
that was the base for this patch.

Cheers,

Angel

Patch

diff --git a/hgext/onsub.py b/hgext/onsub.py
new file mode 100644
--- /dev/null
+++ b/hgext/onsub.py
@@ -0,0 +1,193 @@ 
+# onsub.py - execute commands recursively on subrepositories
+#
+# Copyright 2010, 2011 aragost Trifork
+#
+# This software may be used and distributed according to the terms of
+# the GNU General Public License version 2 or any later version.
+
+import os
+from mercurial.i18n import _
+from mercurial import extensions, subrepo, util
+
+"""execute a command in each subrepository"""
+
+def onsub(ui, repo, *args, **opts):
+    """execute a command in each subrepository
+
+    Executes CMD with the current working directory set to the root of
+    each subrepository. By default, execution stops if CMD returns a
+    non-zero exit code. Use --ignore-errors to override this.
+
+    If a POST-CMD is specified, this will be executed after all
+    subrepositories below the current subrepository has been visited.
+    This corresponds to a post-order traversal of the tree.
+
+    It is an error to specify a POST-CMD together with the
+    --breadth-first flag.
+
+    Use --verbose/-v to print the command being run and the subrepo
+    name for each run of CMD in a subrepo. Alternately, use
+    --print0/-0 to print just the subrepo name followed by a NUL
+    character instead of a newline. This can be useful in combination
+    with :hg:`status --print0`.
+
+    The command has access to the following environment variables:
+
+    ``HG_REPO``:
+        Absolute path to the top-level repository in which the onsub
+        command was executed.
+
+    ``HG_SUBPATH``:
+        Relative path to the current subrepository from the top-level
+        repository.
+
+    ``HG_SUBURL``:
+        URL for the current subrepository as specified in the
+        containing repository's ``.hgsub`` file.
+
+    ``HG_SUBSTATE``:
+        State of the current subrepository as specified in the
+        containing repository's ``.hgsubstate`` file.
+
+    ``HG_SUBTYPE``:
+        The type of the current subrepository (hg, git or svn).
+    """
+
+    # function level "constants" - these won't be modified by the nested functions
+    print0 = opts.get('print0')
+    if opts.get('ignore_errors'):
+        onerr = None
+    else:
+        onerr = util.Abort
+    maxdepth = opts.get('max_depth')
+    precmd = None
+    postcmd = None
+    includeroot = opts.get('root_repo')
+    repotypefilter = opts.get('type')
+
+    def execCmd(sub, cmd, kind):
+        """if sub == None, cmd is executed inside repo; else, inside sub.
+        If cmd == None, do nothing. If cmd == '', do only the print0 (if needed). 
+        Else, do either print0 or the debugging message, then execute the command.
+        kind is the type of the (sub)repo.
+        """
+        if sub == None:
+            envargdict = dict(HG_SUBPATH='.',
+                              HG_SUBURL='.',
+                              HG_SUBSTATE=repo['.'].hex(),
+                              HG_REPO=repo.root,
+                              HG_SUBTYPE=kind)
+            relpath = '.'
+            cmdwd = repo.root
+        else:
+            # subrepo.relpath was renamed to subrepo.subrelpath in
+            # 18b5b6392fcf.
+            if hasattr(subrepo, 'relpath'):
+                relpath = subrepo.relpath(sub)
+            else:
+                relpath = subrepo.subrelpath(sub)
+            envargdict = dict(HG_SUBPATH=relpath,
+                              HG_SUBURL=sub._path,
+                              HG_SUBSTATE=sub._state[1],
+                              HG_REPO=repo.root,
+                              HG_SUBTYPE=kind)
+            cmdwd = os.path.join(repo.root, relpath)
+        if cmd != None and (repotypefilter == '' or repotypefilter == kind):
+            if print0:
+                ui.write(relpath, "\0")
+            if cmd != '':
+                if not print0: ui.note(_("executing '%s' in %s\n") % (cmd, relpath))
+                util.system(cmd, environ=envargdict, cwd=cmdwd, onerr=onerr,
+                            errprefix=_('terminated onsub in %s') % relpath)
+
+    def bfs():
+        """execute precmd in repo.root and in each subrepository, breadth-first"""
+        if includeroot:
+            execCmd(None, precmd, 'hg') 
+        ctx = repo['.']
+        work = [(1, ctx.sub(subpath), ctx.substate[subpath][2]) for subpath in sorted(ctx.substate)]
+        while work:
+            (depth, sub, kind) = work.pop(0)
+            if depth > maxdepth >= 0:
+                continue
+            execCmd(sub, precmd, kind) 
+            if kind == 'hg':
+                rev = sub._state[1]
+                ctx = sub._repo[rev]
+                w = [(depth + 1, ctx.sub(subpath), ctx.substate[subpath][2]) 
+                     for subpath in sorted(ctx.substate)]
+                work.extend(w)
+    
+    def dfs():
+        """execute pre-/postcmd in repo.root and in each subrepository, depth-first"""
+
+        def dfs_rek(depth, sub, kind):
+            if depth > maxdepth >= 0:
+                return
+            execCmd(sub, precmd, kind) 
+            if kind == 'hg':
+                rev = sub._state[1]
+                ctx = sub._repo[rev]
+                for subpath in sorted(ctx.substate):
+                    dfs_rek(depth+1, ctx.sub(subpath), ctx.substate[subpath][2])
+            execCmd(sub, postcmd, kind)
+    
+        ctx = repo['.']
+        work = [(ctx.sub(subpath), ctx.substate[subpath][2]) for subpath in sorted(ctx.substate)]
+        if includeroot:
+            execCmd(None, precmd, 'hg') 
+        for (sub, kind) in work:
+            dfs_rek(1, sub, kind)
+        if includeroot:
+            execCmd(None, postcmd, 'hg') 
+        
+    ### start of main function part ###
+    if len(args) == 2:
+        precmd = args[0]
+        postcmd = args[1]
+        if opts.get('breadth_first') or opts.get('post_order'):
+            raise util.Abort(_("onsub: '-b' and '-p' imply the use of only one command"))
+    elif len(args) == 1:
+        if opts.get('post_order'):
+            precmd = None
+            postcmd = args[0]
+        else:
+            precmd = args[0]
+            postcmd = None
+    elif len(args) == 0:
+        # cmd == '' means only do print0
+        if opts.get('post_order'):
+            precmd = None
+            postcmd = ''
+        else:
+            precmd = ''
+            postcmd = None
+    else:
+        raise util.Abort(_("onsub: at most 2 command arguments required"))
+    if opts.get('post_order') and opts.get('breadth_first'):
+        raise util.Abort(_("onsub: '-b' and '-p' are mutually exclusive"))
+
+    if opts.get('breadth_first'):
+        bfs()
+    else:
+        dfs()
+          
+cmdtable = {
+    "onsub":
+        (onsub,
+         [('b', 'breadth-first', None,
+           _('use breadth-first traversal')),
+          ('p', 'post-order', None,
+           _('use post-order depth-first traversal')),
+          ('', 'root-repo', None,
+           _('include root repository in traversal')),
+          ('', 'max-depth', -1,
+           _('limit recursion to N levels (negative for no limit)'), 'N'),
+          ('', 'ignore-errors', None,
+           _('continue execution despite errors')),
+          ('t', 'type', '',
+           _('the type of repo to filter'), 'TYPE'),
+          ('0', 'print0', None,
+           _('end subrepository names with NUL, for use with xargs'))],
+         _('[-b] [-0] [-t TYPE] [--ignore-errors] CMD [POST-CMD]'))
+}
diff --git a/tests/test-onsub-mixed.t b/tests/test-onsub-mixed.t
new file mode 100644
--- /dev/null
+++ b/tests/test-onsub-mixed.t
@@ -0,0 +1,74 @@ 
+Load extension:
+
+  $ echo "[extensions]" >> $HGRCPATH
+  $ echo "onsub =" >> $HGRCPATH
+
+Create some nicely nested subrepositories with mixed types:
+
+  $ hg init
+  $ for d in a b; do hg init $d; echo "$d = $d" >> .hgsub; done
+  $ git init -q git-i
+  $ cd git-i
+  $ git config core.autocrlf false
+  $ echo something > something
+  $ git add something
+  $ git commit -q -m init
+  $ cd ..
+  $ echo "git-i = [git]$git-i" >> .hgsub
+  $ hg add .hgsub
+
+  $ cd a
+
+  $ git init -q git-j
+  $ cd git-j
+  $ git config core.autocrlf false
+  $ echo something > something
+  $ git add something
+  $ git commit -q -m init
+  $ cd ..
+  $ echo "git-j = [git]git-j" >> .hgsub
+  $ hg add .hgsub
+
+  $ cd ..
+
+  $ hg commit -m init -S
+  committing subrepository a
+
+Test the subrepo type
+
+  $ hg onsub 'echo $HG_SUBPATH = $HG_SUBTYPE'
+  a = hg
+  a/git-j = git
+  b = hg
+  git-i = git
+
+Test the subrepo type including the root repository
+
+  $ hg onsub 'echo $HG_SUBPATH = $HG_SUBTYPE' --root-repo
+  . = hg
+  a = hg
+  a/git-j = git
+  b = hg
+  git-i = git
+
+Test the type filter
+
+  $ hg onsub 'echo $HG_SUBPATH' -t hg
+  a
+  b
+
+  $ hg onsub 'echo $HG_SUBPATH' -t git
+  a/git-j
+  git-i
+
+Test the type filter including the root repository
+
+  $ hg onsub 'echo $HG_SUBPATH' -t hg --root-repo
+  .
+  a
+  b
+
+  $ hg onsub 'echo $HG_SUBPATH' -t git --root-repo
+  a/git-j
+  git-i
+
diff --git a/tests/test-onsub.t b/tests/test-onsub.t
new file mode 100644
--- /dev/null
+++ b/tests/test-onsub.t
@@ -0,0 +1,227 @@ 
+Load extension:
+
+  $ echo "[extensions]" >> $HGRCPATH
+  $ echo "onsub =" >> $HGRCPATH
+
+Check help formatting:
+
+  $ hg help onsub
+  hg onsub [-b] [-0] [-t TYPE] [--ignore-errors] CMD [POST-CMD]
+  
+  execute a command in each subrepository
+  
+      Executes CMD with the current working directory set to the root of each
+      subrepository. By default, execution stops if CMD returns a non-zero exit
+      code. Use --ignore-errors to override this.
+  
+      If a POST-CMD is specified, this will be executed after all
+      subrepositories below the current subrepository has been visited. This
+      corresponds to a post-order traversal of the tree.
+  
+      It is an error to specify a POST-CMD together with the --breadth-first
+      flag.
+  
+      Use --verbose/-v to print the command being run and the subrepo name for
+      each run of CMD in a subrepo. Alternately, use --print0/-0 to print just
+      the subrepo name followed by a NUL character instead of a newline. This
+      can be useful in combination with "hg status --print0".
+  
+      The command has access to the following environment variables:
+  
+      "HG_REPO":
+          Absolute path to the top-level repository in which the onsub command
+          was executed.
+  
+      "HG_SUBPATH":
+          Relative path to the current subrepository from the top-level
+          repository.
+  
+      "HG_SUBURL":
+          URL for the current subrepository as specified in the containing
+          repository's ".hgsub" file.
+  
+      "HG_SUBSTATE":
+          State of the current subrepository as specified in the containing
+          repository's ".hgsubstate" file.
+  
+      "HG_SUBTYPE":
+          The type of the current subrepository (hg, git or svn).
+  
+  options:
+  
+   -b --breadth-first use breadth-first traversal
+   -p --post-order    use post-order depth-first traversal
+      --root-repo     include root repository in traversal
+      --max-depth N   limit recursion to N levels (negative for no limit)
+                      (default: -1)
+      --ignore-errors continue execution despite errors
+   -t --type TYPE     the type of repo to filter
+   -0 --print0        end subrepository names with NUL, for use with xargs
+  
+  (some details hidden, use --verbose to show complete help)
+
+Create some nicely nested subrepositories:
+
+  $ hg init
+  $ for d in a b; do hg init $d; echo "$d = $d" >> .hgsub; done
+  $ hg add .hgsub
+
+  $ cd a
+
+  $ for d in x y; do hg init $d; echo "$d = $d" >> .hgsub; done
+  $ hg add .hgsub
+
+  $ cd y
+  $ for d in r s t; do hg init $d; echo "$d = $d" >> .hgsub; done
+  $ hg add .hgsub
+  $ cd ..
+
+  $ cd ..
+
+  $ cd b
+  $ for d in u v; do hg init $d; echo "$d = $d" >> .hgsub; done
+  $ hg add .hgsub
+  $ cd ..
+
+  $ hg commit -m init -S
+  committing subrepository a
+  committing subrepository a/y
+  committing subrepository b
+
+The default depth-first pre-order traversal:
+
+  $ hg onsub 'echo $HG_SUBPATH'
+  a
+  a/x
+  a/y
+  a/y/r
+  a/y/s
+  a/y/t
+  b
+  b/u
+  b/v
+
+Traversal including the root repository:
+
+  $ hg onsub 'echo $HG_SUBPATH' --root-repo
+  .
+  a
+  a/x
+  a/y
+  a/y/r
+  a/y/s
+  a/y/t
+  b
+  b/u
+  b/v
+
+Depth-first post-order traversal: 
+
+  $ hg onsub 'echo $HG_SUBPATH' --post-order
+  a/x
+  a/y/r
+  a/y/s
+  a/y/t
+  a/y
+  a
+  b/u
+  b/v
+  b
+
+Depth-first pre- and post-order traversal:
+
+  $ hg onsub 'echo pre $HG_SUBPATH'  'echo post $HG_SUBPATH'
+  pre a
+  pre a/x
+  post a/x
+  pre a/y
+  pre a/y/r
+  post a/y/r
+  pre a/y/s
+  post a/y/s
+  pre a/y/t
+  post a/y/t
+  post a/y
+  post a
+  pre b
+  pre b/u
+  post b/u
+  pre b/v
+  post b/v
+  post b
+
+Breadth-first traversal:
+
+  $ hg onsub 'echo $HG_SUBPATH' --breadth-first
+  a
+  b
+  a/x
+  a/y
+  b/u
+  b/v
+  a/y/r
+  a/y/s
+  a/y/t
+
+Limit depth of traversal:
+  $ hg onsub --max-depth 1 'echo $HG_SUBPATH'
+  a
+  b
+  $ hg onsub --max-depth 2 'echo $HG_SUBPATH'
+  a
+  a/x
+  a/y
+  b
+  b/u
+  b/v
+  $ hg onsub --max-depth 2 -b 'echo $HG_SUBPATH'
+  a
+  b
+  a/x
+  a/y
+  b/u
+  b/v
+  $ hg onsub --max-depth 1 -b --root-repo 'echo $HG_SUBPATH'
+  .
+  a
+  b
+
+Test aborting:
+
+  $ hg onsub -v 'test $HG_SUBPATH != "a/y/r"'
+  executing 'test $HG_SUBPATH != "a/y/r"' in a
+  executing 'test $HG_SUBPATH != "a/y/r"' in a/x
+  executing 'test $HG_SUBPATH != "a/y/r"' in a/y
+  executing 'test $HG_SUBPATH != "a/y/r"' in a/y/r
+  abort: terminated onsub in a/y/r: test exited with status 1
+  [255]
+
+Test aborting:
+
+  $ hg onsub -v --ignore-errors false
+  executing 'false' in a
+  executing 'false' in a/x
+  executing 'false' in a/y
+  executing 'false' in a/y/r
+  executing 'false' in a/y/s
+  executing 'false' in a/y/t
+  executing 'false' in b
+  executing 'false' in b/u
+  executing 'false' in b/v
+
+Test --print0:
+
+  $ mv a 'with spaces'
+  $ echo 'with spaces = with spaces' > .hgsub
+  $ echo 'b = b' >> .hgsub
+  $ hg commit -m rename
+  $ hg onsub -0 | xargs -n 1 -0
+  b
+  b/u
+  b/v
+  with spaces
+  with spaces/x
+  with spaces/y
+  with spaces/y/r
+  with spaces/y/s
+  with spaces/y/t