Patchwork [V2] extensions: add onsub extension

login
register
mail settings
Submitter Angel Ezquerra
Date Oct. 12, 2014, 2:09 p.m.
Message ID <76a648d782cfadf1dac3.1413122955@Angels-MacBook-Pro.local>
Download mbox | patch
Permalink /patch/6234/
State Deferred
Headers show

Comments

Angel Ezquerra - Oct. 12, 2014, 2:09 p.m.
# HG changeset patch
# User Angel Ezquerra <angel.ezquerra@gmail.com>
# Date 1409641307 -7200
#      Tue Sep 02 09:01:47 2014 +0200
# Node ID 76a648d782cfadf1dac3dbc194a8ffc74c385cb5
# Parent  1533e642262de32c5a2445789710f49237019fd6
extensions: add onsub extension

The onsub extension lets you execute a command in each subrepository. This is
useful when working with a repository that contains a large number of
subrepositories. For example, it can be used for quickly updating all
subrepositories:

  $ hg onsub 'hg pull --update'

This extension was originally written by Martin Geisler for aragost Trifork. The
original repository is on bitbucket (https://bitbucket.org/lantiq/onsub/).
The extension is quite simple (it has not required any changes since 2012, and
it continues working fine to this day). The extension has a couple of tests.

I discussed the inclusion of this extension with Martin Geisler, the original
author of the extension, during the Munich 3.2 sprint. He agreed to the
inclusion of the extension, which I will try to maintain from now on.

Also note that this extension has been included with TortoiseHg for several
major releases.

The code for the extension that included on this patch is identical to the
latest version of the extension on its bitbucket repository (30bdbd5). I just
had to make a few small changes to the tests:

1. some changes were needed because the extension is now in the hgext folder,
2. adapt to a small change to the --verbose hint on the extension help
3. adapt to a change on the output of mercurial's commit --subrepos command

None of these changes is related to the functionality of the extension itself,
which as I said has not changed in a couple of years.
Yuya Nishihara - Oct. 12, 2014, 3:40 p.m.
On Sun, 12 Oct 2014 16:09:15 +0200, Angel Ezquerra wrote:
> # HG changeset patch
> # User Angel Ezquerra <angel.ezquerra@gmail.com>
> # Date 1409641307 -7200
> #      Tue Sep 02 09:01:47 2014 +0200
> # Node ID 76a648d782cfadf1dac3dbc194a8ffc74c385cb5
> # Parent  1533e642262de32c5a2445789710f49237019fd6
> extensions: add onsub extension

> +    def execCmd(sub, cmd, kind):
> +        """if sub == None, cmd is executed inside repo; else, inside sub.
> +        If cmd == None, do nothing. If cmd == '', do only the print0 (if needed). 
> +        Else, do either print0 or the debugging message, then execute the command.
> +        kind is the type of the (sub)repo.
> +        """
> +        if sub == None:
> +            envargdict = dict(HG_SUBPATH='.',
> +                              HG_SUBURL='.',
> +                              HG_SUBSTATE=repo['.'].hex(),
> +                              HG_REPO=repo.root,
> +                              HG_SUBTYPE=kind)
> +            relpath = '.'
> +            cmdwd = repo.root
> +        else:
> +            # subrepo.relpath was renamed to subrepo.subrelpath in
> +            # 18b5b6392fcf.
> +            if hasattr(subrepo, 'relpath'):
> +                relpath = subrepo.relpath(sub)
> +            else:
> +                relpath = subrepo.subrelpath(sub)
> +            envargdict = dict(HG_SUBPATH=relpath,
> +                              HG_SUBURL=sub._path,
> +                              HG_SUBSTATE=sub._state[1],
> +                              HG_REPO=repo.root,
> +                              HG_SUBTYPE=kind)
> +            cmdwd = os.path.join(repo.root, relpath)
> +        if cmd != None and (repotypefilter == '' or repotypefilter == kind):
> +            if print0:
> +                ui.write(relpath, "\0")
> +            if cmd != '':
> +                if not print0: ui.note(_("executing '%s' in %s\n") % (cmd, relpath))
> +                util.system(cmd, environ=envargdict, cwd=cmdwd, onerr=onerr,
> +                            errprefix=_('terminated onsub in %s') % relpath)

I didn't look through the onsub extension, but this util.system() call will
break the command-server channel.

https://bitbucket.org/tortoisehg/thg/issue/3924/
(no response yet, as usual)

Regards,
Angel Ezquerra - Oct. 12, 2014, 6:03 p.m.
On Sun, Oct 12, 2014 at 5:40 PM, Yuya Nishihara <yuya@tcha.org> wrote:
> On Sun, 12 Oct 2014 16:09:15 +0200, Angel Ezquerra wrote:
>> # HG changeset patch
>> # User Angel Ezquerra <angel.ezquerra@gmail.com>
>> # Date 1409641307 -7200
>> #      Tue Sep 02 09:01:47 2014 +0200
>> # Node ID 76a648d782cfadf1dac3dbc194a8ffc74c385cb5
>> # Parent  1533e642262de32c5a2445789710f49237019fd6
>> extensions: add onsub extension
>
>> +    def execCmd(sub, cmd, kind):
>> +        """if sub == None, cmd is executed inside repo; else, inside sub.
>> +        If cmd == None, do nothing. If cmd == '', do only the print0 (if needed).
>> +        Else, do either print0 or the debugging message, then execute the command.
>> +        kind is the type of the (sub)repo.
>> +        """
>> +        if sub == None:
>> +            envargdict = dict(HG_SUBPATH='.',
>> +                              HG_SUBURL='.',
>> +                              HG_SUBSTATE=repo['.'].hex(),
>> +                              HG_REPO=repo.root,
>> +                              HG_SUBTYPE=kind)
>> +            relpath = '.'
>> +            cmdwd = repo.root
>> +        else:
>> +            # subrepo.relpath was renamed to subrepo.subrelpath in
>> +            # 18b5b6392fcf.
>> +            if hasattr(subrepo, 'relpath'):
>> +                relpath = subrepo.relpath(sub)
>> +            else:
>> +                relpath = subrepo.subrelpath(sub)
>> +            envargdict = dict(HG_SUBPATH=relpath,
>> +                              HG_SUBURL=sub._path,
>> +                              HG_SUBSTATE=sub._state[1],
>> +                              HG_REPO=repo.root,
>> +                              HG_SUBTYPE=kind)
>> +            cmdwd = os.path.join(repo.root, relpath)
>> +        if cmd != None and (repotypefilter == '' or repotypefilter == kind):
>> +            if print0:
>> +                ui.write(relpath, "\0")
>> +            if cmd != '':
>> +                if not print0: ui.note(_("executing '%s' in %s\n") % (cmd, relpath))
>> +                util.system(cmd, environ=envargdict, cwd=cmdwd, onerr=onerr,
>> +                            errprefix=_('terminated onsub in %s') % relpath)
>
> I didn't look through the onsub extension, but this util.system() call will
> break the command-server channel.
>
> https://bitbucket.org/tortoisehg/thg/issue/3924/
> (no response yet, as usual)

Wow, that seems easy to get wrong! I can fix it following your
suggestion in that bitbucket issue and send it again.

I did a quick search of util.system calls in mercurial's code base and
I found one in sshpeer.py, line 55:

res = util.system(cmd)

There is another one in the last line of hgk.py.

Should those be changed then?

Cheers,

Angel
Yuya Nishihara - Oct. 13, 2014, 2:57 p.m.
On Sun, 12 Oct 2014 20:03:18 +0200, Angel Ezquerra wrote:
> On Sun, Oct 12, 2014 at 5:40 PM, Yuya Nishihara <yuya@tcha.org> wrote:
> > On Sun, 12 Oct 2014 16:09:15 +0200, Angel Ezquerra wrote:
> >> +                util.system(cmd, environ=envargdict, cwd=cmdwd, onerr=onerr,
> >> +                            errprefix=_('terminated onsub in %s') % relpath)
> >
> > I didn't look through the onsub extension, but this util.system() call will
> > break the command-server channel.
> >
> > https://bitbucket.org/tortoisehg/thg/issue/3924/
> > (no response yet, as usual)
> 
> Wow, that seems easy to get wrong! I can fix it following your
> suggestion in that bitbucket issue and send it again.
> 
> I did a quick search of util.system calls in mercurial's code base and
> I found one in sshpeer.py, line 55:
> 
> res = util.system(cmd)
>
> There is another one in the last line of hgk.py.
>
> Should those be changed then?

Maybe yes.  Though it rarely happens, it's possible to make remote "hg init"
say something to stdout, for example,

    [hooks]
    pre-init = echo 'hi'

I have no idea whether hgk should be usable in commandserver, but out=ui.fout
have no bad effect on general use.

Regards,

Patch

diff -r 1533e642262d -r 76a648d782cf hgext/onsub.py
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/hgext/onsub.py	Tue Sep 02 09:01:47 2014 +0200
@@ -0,0 +1,193 @@ 
+# onsub.py - execute commands recursively on subrepositories
+#
+# Copyright 2010, 2011 aragost Trifork
+#
+# This software may be used and distributed according to the terms of
+# the GNU General Public License version 2 or any later version.
+
+import os
+from mercurial.i18n import _
+from mercurial import extensions, subrepo, util
+
+"""execute a command in each subrepository"""
+
+def onsub(ui, repo, *args, **opts):
+    """execute a command in each subrepository
+
+    Executes CMD with the current working directory set to the root of
+    each subrepository. By default, execution stops if CMD returns a
+    non-zero exit code. Use --ignore-errors to override this.
+
+    If a POST-CMD is specified, this will be executed after all
+    subrepositories below the current subrepository has been visited.
+    This corresponds to a post-order traversal of the tree.
+
+    It is an error to specify a POST-CMD together with the
+    --breadth-first flag.
+
+    Use --verbose/-v to print the command being run and the subrepo
+    name for each run of CMD in a subrepo. Alternately, use
+    --print0/-0 to print just the subrepo name followed by a NUL
+    character instead of a newline. This can be useful in combination
+    with :hg:`status --print0`.
+
+    The command has access to the following environment variables:
+
+    ``HG_REPO``:
+        Absolute path to the top-level repository in which the onsub
+        command was executed.
+
+    ``HG_SUBPATH``:
+        Relative path to the current subrepository from the top-level
+        repository.
+
+    ``HG_SUBURL``:
+        URL for the current subrepository as specified in the
+        containing repository's ``.hgsub`` file.
+
+    ``HG_SUBSTATE``:
+        State of the current subrepository as specified in the
+        containing repository's ``.hgsubstate`` file.
+
+    ``HG_SUBTYPE``:
+        The type of the current subrepository (hg, git or svn).
+    """
+
+    # function level "constants" - these won't be modified by the nested functions
+    print0 = opts.get('print0')
+    if opts.get('ignore_errors'):
+        onerr = None
+    else:
+        onerr = util.Abort
+    maxdepth = opts.get('max_depth')
+    precmd = None
+    postcmd = None
+    includeroot = opts.get('root_repo')
+    repotypefilter = opts.get('type')
+
+    def execCmd(sub, cmd, kind):
+        """if sub == None, cmd is executed inside repo; else, inside sub.
+        If cmd == None, do nothing. If cmd == '', do only the print0 (if needed). 
+        Else, do either print0 or the debugging message, then execute the command.
+        kind is the type of the (sub)repo.
+        """
+        if sub == None:
+            envargdict = dict(HG_SUBPATH='.',
+                              HG_SUBURL='.',
+                              HG_SUBSTATE=repo['.'].hex(),
+                              HG_REPO=repo.root,
+                              HG_SUBTYPE=kind)
+            relpath = '.'
+            cmdwd = repo.root
+        else:
+            # subrepo.relpath was renamed to subrepo.subrelpath in
+            # 18b5b6392fcf.
+            if hasattr(subrepo, 'relpath'):
+                relpath = subrepo.relpath(sub)
+            else:
+                relpath = subrepo.subrelpath(sub)
+            envargdict = dict(HG_SUBPATH=relpath,
+                              HG_SUBURL=sub._path,
+                              HG_SUBSTATE=sub._state[1],
+                              HG_REPO=repo.root,
+                              HG_SUBTYPE=kind)
+            cmdwd = os.path.join(repo.root, relpath)
+        if cmd != None and (repotypefilter == '' or repotypefilter == kind):
+            if print0:
+                ui.write(relpath, "\0")
+            if cmd != '':
+                if not print0: ui.note(_("executing '%s' in %s\n") % (cmd, relpath))
+                util.system(cmd, environ=envargdict, cwd=cmdwd, onerr=onerr,
+                            errprefix=_('terminated onsub in %s') % relpath)
+
+    def bfs():
+        """execute precmd in repo.root and in each subrepository, breadth-first"""
+        if includeroot:
+            execCmd(None, precmd, 'hg') 
+        ctx = repo['.']
+        work = [(1, ctx.sub(subpath), ctx.substate[subpath][2]) for subpath in sorted(ctx.substate)]
+        while work:
+            (depth, sub, kind) = work.pop(0)
+            if depth > maxdepth >= 0:
+                continue
+            execCmd(sub, precmd, kind) 
+            if kind == 'hg':
+                rev = sub._state[1]
+                ctx = sub._repo[rev]
+                w = [(depth + 1, ctx.sub(subpath), ctx.substate[subpath][2]) 
+                     for subpath in sorted(ctx.substate)]
+                work.extend(w)
+    
+    def dfs():
+        """execute pre-/postcmd in repo.root and in each subrepository, depth-first"""
+
+        def dfs_rek(depth, sub, kind):
+            if depth > maxdepth >= 0:
+                return
+            execCmd(sub, precmd, kind) 
+            if kind == 'hg':
+                rev = sub._state[1]
+                ctx = sub._repo[rev]
+                for subpath in sorted(ctx.substate):
+                    dfs_rek(depth+1, ctx.sub(subpath), ctx.substate[subpath][2])
+            execCmd(sub, postcmd, kind)
+    
+        ctx = repo['.']
+        work = [(ctx.sub(subpath), ctx.substate[subpath][2]) for subpath in sorted(ctx.substate)]
+        if includeroot:
+            execCmd(None, precmd, 'hg') 
+        for (sub, kind) in work:
+            dfs_rek(1, sub, kind)
+        if includeroot:
+            execCmd(None, postcmd, 'hg') 
+        
+    ### start of main function part ###
+    if len(args) == 2:
+        precmd = args[0]
+        postcmd = args[1]
+        if opts.get('breadth_first') or opts.get('post_order'):
+            raise util.Abort(_("onsub: '-b' and '-p' imply the use of only one command"))
+    elif len(args) == 1:
+        if opts.get('post_order'):
+            precmd = None
+            postcmd = args[0]
+        else:
+            precmd = args[0]
+            postcmd = None
+    elif len(args) == 0:
+        # cmd == '' means only do print0
+        if opts.get('post_order'):
+            precmd = None
+            postcmd = ''
+        else:
+            precmd = ''
+            postcmd = None
+    else:
+        raise util.Abort(_("onsub: at most 2 command arguments required"))
+    if opts.get('post_order') and opts.get('breadth_first'):
+        raise util.Abort(_("onsub: '-b' and '-p' are mutually exclusive"))
+
+    if opts.get('breadth_first'):
+        bfs()
+    else:
+        dfs()
+          
+cmdtable = {
+    "onsub":
+        (onsub,
+         [('b', 'breadth-first', None,
+           _('use breadth-first traversal')),
+          ('p', 'post-order', None,
+           _('use post-order depth-first traversal')),
+          ('', 'root-repo', None,
+           _('include root repository in traversal')),
+          ('', 'max-depth', -1,
+           _('limit recursion to N levels (negative for no limit)'), 'N'),
+          ('', 'ignore-errors', None,
+           _('continue execution despite errors')),
+          ('t', 'type', '',
+           _('the type of repo to filter'), 'TYPE'),
+          ('0', 'print0', None,
+           _('end subrepository names with NUL, for use with xargs'))],
+         _('[-b] [-0] [-t TYPE] [--ignore-errors] CMD [POST-CMD]'))
+}
diff -r 1533e642262d -r 76a648d782cf tests/test-onsub-mixed.t
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/tests/test-onsub-mixed.t	Tue Sep 02 09:01:47 2014 +0200
@@ -0,0 +1,74 @@ 
+Load extension:
+
+  $ echo "[extensions]" >> $HGRCPATH
+  $ echo "onsub =" >> $HGRCPATH
+
+Create some nicely nested subrepositories with mixed types:
+
+  $ hg init
+  $ for d in a b; do hg init $d; echo "$d = $d" >> .hgsub; done
+  $ git init -q git-i
+  $ cd git-i
+  $ git config core.autocrlf false
+  $ echo something > something
+  $ git add something
+  $ git commit -q -m init
+  $ cd ..
+  $ echo "git-i = [git]$git-i" >> .hgsub
+  $ hg add .hgsub
+
+  $ cd a
+
+  $ git init -q git-j
+  $ cd git-j
+  $ git config core.autocrlf false
+  $ echo something > something
+  $ git add something
+  $ git commit -q -m init
+  $ cd ..
+  $ echo "git-j = [git]git-j" >> .hgsub
+  $ hg add .hgsub
+
+  $ cd ..
+
+  $ hg commit -m init -S
+  committing subrepository a
+
+Test the subrepo type
+
+  $ hg onsub 'echo $HG_SUBPATH = $HG_SUBTYPE'
+  a = hg
+  a/git-j = git
+  b = hg
+  git-i = git
+
+Test the subrepo type including the root repository
+
+  $ hg onsub 'echo $HG_SUBPATH = $HG_SUBTYPE' --root-repo
+  . = hg
+  a = hg
+  a/git-j = git
+  b = hg
+  git-i = git
+
+Test the type filter
+
+  $ hg onsub 'echo $HG_SUBPATH' -t hg
+  a
+  b
+
+  $ hg onsub 'echo $HG_SUBPATH' -t git
+  a/git-j
+  git-i
+
+Test the type filter including the root repository
+
+  $ hg onsub 'echo $HG_SUBPATH' -t hg --root-repo
+  .
+  a
+  b
+
+  $ hg onsub 'echo $HG_SUBPATH' -t git --root-repo
+  a/git-j
+  git-i
+
diff -r 1533e642262d -r 76a648d782cf tests/test-onsub.t
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/tests/test-onsub.t	Tue Sep 02 09:01:47 2014 +0200
@@ -0,0 +1,227 @@ 
+Load extension:
+
+  $ echo "[extensions]" >> $HGRCPATH
+  $ echo "onsub =" >> $HGRCPATH
+
+Check help formatting:
+
+  $ hg help onsub
+  hg onsub [-b] [-0] [-t TYPE] [--ignore-errors] CMD [POST-CMD]
+  
+  execute a command in each subrepository
+  
+      Executes CMD with the current working directory set to the root of each
+      subrepository. By default, execution stops if CMD returns a non-zero exit
+      code. Use --ignore-errors to override this.
+  
+      If a POST-CMD is specified, this will be executed after all
+      subrepositories below the current subrepository has been visited. This
+      corresponds to a post-order traversal of the tree.
+  
+      It is an error to specify a POST-CMD together with the --breadth-first
+      flag.
+  
+      Use --verbose/-v to print the command being run and the subrepo name for
+      each run of CMD in a subrepo. Alternately, use --print0/-0 to print just
+      the subrepo name followed by a NUL character instead of a newline. This
+      can be useful in combination with "hg status --print0".
+  
+      The command has access to the following environment variables:
+  
+      "HG_REPO":
+          Absolute path to the top-level repository in which the onsub command
+          was executed.
+  
+      "HG_SUBPATH":
+          Relative path to the current subrepository from the top-level
+          repository.
+  
+      "HG_SUBURL":
+          URL for the current subrepository as specified in the containing
+          repository's ".hgsub" file.
+  
+      "HG_SUBSTATE":
+          State of the current subrepository as specified in the containing
+          repository's ".hgsubstate" file.
+  
+      "HG_SUBTYPE":
+          The type of the current subrepository (hg, git or svn).
+  
+  options:
+  
+   -b --breadth-first use breadth-first traversal
+   -p --post-order    use post-order depth-first traversal
+      --root-repo     include root repository in traversal
+      --max-depth N   limit recursion to N levels (negative for no limit)
+                      (default: -1)
+      --ignore-errors continue execution despite errors
+   -t --type TYPE     the type of repo to filter
+   -0 --print0        end subrepository names with NUL, for use with xargs
+  
+  (some details hidden, use --verbose to show complete help)
+
+Create some nicely nested subrepositories:
+
+  $ hg init
+  $ for d in a b; do hg init $d; echo "$d = $d" >> .hgsub; done
+  $ hg add .hgsub
+
+  $ cd a
+
+  $ for d in x y; do hg init $d; echo "$d = $d" >> .hgsub; done
+  $ hg add .hgsub
+
+  $ cd y
+  $ for d in r s t; do hg init $d; echo "$d = $d" >> .hgsub; done
+  $ hg add .hgsub
+  $ cd ..
+
+  $ cd ..
+
+  $ cd b
+  $ for d in u v; do hg init $d; echo "$d = $d" >> .hgsub; done
+  $ hg add .hgsub
+  $ cd ..
+
+  $ hg commit -m init -S
+  committing subrepository a
+  committing subrepository a/y
+  committing subrepository b
+
+The default depth-first pre-order traversal:
+
+  $ hg onsub 'echo $HG_SUBPATH'
+  a
+  a/x
+  a/y
+  a/y/r
+  a/y/s
+  a/y/t
+  b
+  b/u
+  b/v
+
+Traversal including the root repository:
+
+  $ hg onsub 'echo $HG_SUBPATH' --root-repo
+  .
+  a
+  a/x
+  a/y
+  a/y/r
+  a/y/s
+  a/y/t
+  b
+  b/u
+  b/v
+
+Depth-first post-order traversal: 
+
+  $ hg onsub 'echo $HG_SUBPATH' --post-order
+  a/x
+  a/y/r
+  a/y/s
+  a/y/t
+  a/y
+  a
+  b/u
+  b/v
+  b
+
+Depth-first pre- and post-order traversal:
+
+  $ hg onsub 'echo pre $HG_SUBPATH'  'echo post $HG_SUBPATH'
+  pre a
+  pre a/x
+  post a/x
+  pre a/y
+  pre a/y/r
+  post a/y/r
+  pre a/y/s
+  post a/y/s
+  pre a/y/t
+  post a/y/t
+  post a/y
+  post a
+  pre b
+  pre b/u
+  post b/u
+  pre b/v
+  post b/v
+  post b
+
+Breadth-first traversal:
+
+  $ hg onsub 'echo $HG_SUBPATH' --breadth-first
+  a
+  b
+  a/x
+  a/y
+  b/u
+  b/v
+  a/y/r
+  a/y/s
+  a/y/t
+
+Limit depth of traversal:
+  $ hg onsub --max-depth 1 'echo $HG_SUBPATH'
+  a
+  b
+  $ hg onsub --max-depth 2 'echo $HG_SUBPATH'
+  a
+  a/x
+  a/y
+  b
+  b/u
+  b/v
+  $ hg onsub --max-depth 2 -b 'echo $HG_SUBPATH'
+  a
+  b
+  a/x
+  a/y
+  b/u
+  b/v
+  $ hg onsub --max-depth 1 -b --root-repo 'echo $HG_SUBPATH'
+  .
+  a
+  b
+
+Test aborting:
+
+  $ hg onsub -v 'test $HG_SUBPATH != "a/y/r"'
+  executing 'test $HG_SUBPATH != "a/y/r"' in a
+  executing 'test $HG_SUBPATH != "a/y/r"' in a/x
+  executing 'test $HG_SUBPATH != "a/y/r"' in a/y
+  executing 'test $HG_SUBPATH != "a/y/r"' in a/y/r
+  abort: terminated onsub in a/y/r: test exited with status 1
+  [255]
+
+Test aborting:
+
+  $ hg onsub -v --ignore-errors false
+  executing 'false' in a
+  executing 'false' in a/x
+  executing 'false' in a/y
+  executing 'false' in a/y/r
+  executing 'false' in a/y/s
+  executing 'false' in a/y/t
+  executing 'false' in b
+  executing 'false' in b/u
+  executing 'false' in b/v
+
+Test --print0:
+
+  $ mv a 'with spaces'
+  $ echo 'with spaces = with spaces' > .hgsub
+  $ echo 'b = b' >> .hgsub
+  $ hg commit -m rename
+  $ hg onsub -0 | xargs -n 1 -0
+  b
+  b/u
+  b/v
+  with spaces
+  with spaces/x
+  with spaces/y
+  with spaces/y/r
+  with spaces/y/s
+  with spaces/y/t