Patchwork [V4] serve: add support for Mercurial subrepositories

# HG changeset patch
# User Matt Harbison <matt_harbison@yahoo.com>
# Date 1492293940 14400
#      Sat Apr 15 18:05:40 2017 -0400
# Node ID 2804acb18b59dec98abb216d813cc0e3ecec4e14
# Parent  3fd50d5f9314a96f43eb73480763f224c4d05831
serve: add support for Mercurial subrepositories

I've been using `hg serve --web-conf ...` with a simple '/=projects/**' [paths]
configuration for awhile without issue.  Let's ditch the need for the manual
configuration in this case, and limit the repos served to the actual subrepos.

This doesn't attempt to handle the case where a new subrepo appears while the
server is running.  That could probably be handled with a hook if somebody wants
it.  But it's such a rare case, it probably doesn't matter for the temporary
serves.

The main repo is served at '/', just like a repository without subrepos.  I'm
not sure why the duplicate 'adding ...' lines appear on Linux.  They don't
appear on Windows (see 594dd384803c), so they are optional.

Subrepositories that are configured with '../path' or absolute paths are not
cloneable from the server.  (They aren't cloneable locally either, unless they
also exist at their configured source, perhaps via the share extension.)  They
are still served, so that they can be browsed, or cloned individually.  If we
care about that cloning someday, we can probably just add the extra entries to
the webconf dictionary.  Even if the entries use '../' to escape the root, only
the related subrepositories would end up in the dictionary.

This depends on the hosting at '/' patch submitted earlier today.

> On Apr 15, 2017, at 7:00 PM, Matt Harbison <mharbison72@gmail.com> wrote:
> 
> # HG changeset patch
> # User Matt Harbison <matt_harbison@yahoo.com>
> # Date 1492293940 14400
> #      Sat Apr 15 18:05:40 2017 -0400
> # Node ID 2804acb18b59dec98abb216d813cc0e3ecec4e14
> # Parent  3fd50d5f9314a96f43eb73480763f224c4d05831
> serve: add support for Mercurial subrepositories
> 
> I've been using `hg serve --web-conf ...` with a simple '/=projects/**' [paths]
> configuration for awhile without issue.  Let's ditch the need for the manual
> configuration in this case, and limit the repos served to the actual subrepos.
> 
> This doesn't attempt to handle the case where a new subrepo appears while the
> server is running.  That could probably be handled with a hook if somebody wants
> it.  But it's such a rare case, it probably doesn't matter for the temporary
> serves.
> 
> The main repo is served at '/', just like a repository without subrepos.  I'm
> not sure why the duplicate 'adding ...' lines appear on Linux.  They don't
> appear on Windows (see 594dd384803c), so they are optional.
> 
> Subrepositories that are configured with '../path' or absolute paths are not
> cloneable from the server.  (They aren't cloneable locally either, unless they
> also exist at their configured source, perhaps via the share extension.)  They
> are still served, so that they can be browsed, or cloned individually.  If we
> care about that cloning someday, we can probably just add the extra entries to
> the webconf dictionary.  Even if the entries use '../' to escape the root, only
> the related subrepositories would end up in the dictionary.
> 
> diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
> --- a/mercurial/cmdutil.py
> +++ b/mercurial/cmdutil.py
> @@ -2290,6 +2290,15 @@
>         bad.extend(f for f in rejected if f in match.files())
>     return bad
> 
> +def addwebdirpath(repo, serverpath, webconf):
> +    webconf[serverpath] = repo.root
> +    repo.ui.debug('adding %s = %s\n' % (serverpath, repo.root))
> +
> +    for r in repo.revs('filelog("path:.hgsub")'):
> +        ctx = repo[r]
> +        for subpath in ctx.substate:
> +            ctx.sub(subpath).addwebdirpath(serverpath, webconf)
> +
> def forget(ui, repo, match, prefix, explicitonly):
>     join = lambda f: os.path.join(prefix, f)
>     bad = []
> diff --git a/mercurial/commands.py b/mercurial/commands.py
> --- a/mercurial/commands.py
> +++ b/mercurial/commands.py
> @@ -4619,7 +4619,8 @@
>     ('t', 'templates', '', _('web templates to use'), _('TEMPLATE')),
>     ('', 'style', '', _('template style to use'), _('STYLE')),
>     ('6', 'ipv6', None, _('use IPv6 in addition to IPv4')),
> -    ('', 'certificate', '', _('SSL certificate file'), _('FILE'))],
> +    ('', 'certificate', '', _('SSL certificate file'), _('FILE'))]
> +     + subrepoopts,
>     _('[OPTION]...'),
>     optionalrepo=True)
> def serve(ui, repo, **opts):
> diff --git a/mercurial/help/subrepos.txt b/mercurial/help/subrepos.txt
> --- a/mercurial/help/subrepos.txt
> +++ b/mercurial/help/subrepos.txt
> @@ -136,6 +136,10 @@
>     subrepository changes are available when referenced by top-level
>     repositories.  Push is a no-op for Subversion subrepositories.
> 
> +:serve: serve does not recurse into subrepositories unless
> +    -S/--subrepos is specified.  Git and Subversion subrepositories
> +    are currently silently ignored.
> +
> :status: status does not recurse into subrepositories unless
>     -S/--subrepos is specified. Subrepository changes are displayed as
>     regular Mercurial changes on the subrepository
> diff --git a/mercurial/server.py b/mercurial/server.py
> --- a/mercurial/server.py
> +++ b/mercurial/server.py
> @@ -15,6 +15,7 @@
> 
> from . import (
>     chgserver,
> +    cmdutil,
>     commandserver,
>     error,
>     hgweb,
> @@ -130,11 +131,22 @@
>         baseui = ui
>     webconf = opts.get('web_conf') or opts.get('webdir_conf')
>     if webconf:
> +        if opts.get('subrepos'):
> +            raise error.Abort(_('--web-conf cannot be used with --subrepos'))
> +
>         # load server settings (e.g. web.port) to "copied" ui, which allows
>         # hgwebdir to reload webconf cleanly
>         servui = ui.copy()
>         servui.readconfig(webconf, sections=['web'])
>         alluis.add(servui)
> +    elif opts.get('subrepos'):
> +        servui = ui
> +
> +        # If repo is None, hgweb.createapp() already raises a proper abort
> +        # message as long as webconf is None.
> +        if repo:
> +            webconf = dict()
> +            cmdutil.addwebdirpath(repo, "", webconf)
>     else:
>         servui = ui
> 
> diff --git a/mercurial/subrepo.py b/mercurial/subrepo.py
> --- a/mercurial/subrepo.py
> +++ b/mercurial/subrepo.py
> @@ -444,6 +444,15 @@
>         self._ctx = ctx
>         self._path = path
> 
> +    def addwebdirpath(self, serverpath, webconf):
> +        """Add the hgwebdir entries for this subrepo, and any of its subrepos.
> +
> +        ``serverpath`` is the path component of the URL for this repo.
> +
> +        ``webconf`` is the dictionary of hgwebdir entries.
> +        """
> +        pass
> +
>     def storeclean(self, path):
>         """
>         returns true if the repository has not changed since it was last
> @@ -651,6 +660,10 @@
>         self.ui.setconfig('ui', '_usedassubrepo', 'True', 'subrepo')
>         self._initrepo(r, state[0], create)
> 
> +    @annotatesubrepoerror
> +    def addwebdirpath(self, serverpath, webconf):
> +        cmdutil.addwebdirpath(self._repo, subrelpath(self), webconf)
> +
>     def storeclean(self, path):
>         with self._repo.lock():
>             return self._storeclean(path)
> diff --git a/tests/test-completion.t b/tests/test-completion.t
> --- a/tests/test-completion.t
> +++ b/tests/test-completion.t
> @@ -184,6 +184,7 @@
>   --repository
>   --stdio
>   --style
> +  --subrepos
>   --templates
>   --time
>   --traceback
> @@ -194,6 +195,7 @@
>   -A
>   -E
>   -R
> +  -S
>   -a
>   -d
>   -h
> @@ -225,7 +227,7 @@
>   pull: update, force, rev, bookmark, branch, ssh, remotecmd, insecure
>   push: force, rev, bookmark, branch, new-branch, ssh, remotecmd, insecure
>   remove: after, force, subrepos, include, exclude
> -  serve: accesslog, daemon, daemon-postexec, errorlog, port, address, prefix, name, web-conf, webdir-conf, pid-file, stdio, cmdserver, templates, style, ipv6, certificate
> +  serve: accesslog, daemon, daemon-postexec, errorlog, port, address, prefix, name, web-conf, webdir-conf, pid-file, stdio, cmdserver, templates, style, ipv6, certificate, subrepos
>   status: all, modified, added, removed, deleted, clean, unknown, ignored, no-status, copies, print0, rev, change, include, exclude, subrepos, template
>   summary: remote
>   update: clean, check, merge, date, rev, tool
> diff --git a/tests/test-subrepo-deep-nested-change.t b/tests/test-subrepo-deep-nested-change.t
> --- a/tests/test-subrepo-deep-nested-change.t
> +++ b/tests/test-subrepo-deep-nested-change.t
> @@ -73,6 +73,43 @@
>   adding main/main (glob)
>   $ hg commit -R main -m "main import"
> 
> +#if serve
> +
> +Unfortunately, subrepos not at their nominal location cannot be cloned.  But
> +they are still served from their location within the local repository.  The only
> +reason why 'main' can be cloned via the filesystem is because 'sub1' and 'sub2'
> +are also available as siblings of 'main'.
> +
> +  $ hg serve -R main --debug -S -p $HGPORT -d --pid-file=hg1.pid -E error.log -A access.log
> +  adding  = $TESTTMP/main (glob)
> +  adding sub1 = $TESTTMP/main/sub1 (glob)
> +  adding sub1/sub2 = $TESTTMP/main/sub1/sub2 (glob)
> +  listening at http://*:$HGPORT/ (bound to *:$HGPORT) (glob) (?)
> +  adding  = $TESTTMP/main (glob) (?)
> +  adding sub1 = $TESTTMP/main/sub1 (glob) (?)
> +  adding sub1/sub2 = $TESTTMP/main/sub1/sub2 (glob) (?)
> +  $ cat hg1.pid >> $DAEMON_PIDS
> +
> +  $ hg clone http://localhost:$HGPORT httpclone --config progress.disable=True
> +  requesting all changes
> +  adding changesets
> +  adding manifests
> +  adding file changes
> +  added 1 changesets with 3 changes to 3 files
> +  updating to branch default
> +  abort: HTTP Error 404: Not Found
> +  [255]
> +
> +  $ cat access.log
> +  * "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
> +  * "GET /?cmd=batch HTTP/1.1" 200 - * (glob)
> +  * "GET /?cmd=getbundle HTTP/1.1" 200 - * (glob)
> +  * "GET /../sub1?cmd=capabilities HTTP/1.1" 404 - (glob)
> +
> +  $ killdaemons.py
> +  $ rm hg1.pid error.log access.log
> +#endif
> +
> Cleaning both repositories, just as a clone -U
> 
>   $ hg up -C -R sub2 null
> diff --git a/tests/test-subrepo-recursion.t b/tests/test-subrepo-recursion.t
> --- a/tests/test-subrepo-recursion.t
> +++ b/tests/test-subrepo-recursion.t
> @@ -251,6 +251,60 @@
>    z1
>   +z2
> 
> +#if serve
> +  $ cd ..
> +  $ hg serve -R repo --debug -S -p $HGPORT -d --pid-file=hg1.pid -E error.log -A access.log
> +  adding  = $TESTTMP/repo (glob)
> +  adding foo = $TESTTMP/repo/foo (glob)
> +  adding foo/bar = $TESTTMP/repo/foo/bar (glob)
> +  listening at http://*:$HGPORT/ (bound to *:$HGPORT) (glob) (?)
> +  adding  = $TESTTMP/repo (glob) (?)
> +  adding foo = $TESTTMP/repo/foo (glob) (?)
> +  adding foo/bar = $TESTTMP/repo/foo/bar (glob) (?)
> +  $ cat hg1.pid >> $DAEMON_PIDS
> +
> +  $ hg clone http://localhost:$HGPORT clone  --config progress.disable=True
> +  requesting all changes
> +  adding changesets
> +  adding manifests
> +  adding file changes
> +  added 3 changesets with 5 changes to 3 files
> +  updating to branch default
> +  cloning subrepo foo from http://localhost:$HGPORT/foo
> +  requesting all changes
> +  adding changesets
> +  adding manifests
> +  adding file changes
> +  added 4 changesets with 7 changes to 3 files
> +  cloning subrepo foo/bar from http://localhost:$HGPORT/foo/bar (glob)
> +  requesting all changes
> +  adding changesets
> +  adding manifests
> +  adding file changes
> +  added 3 changesets with 3 changes to 1 files
> +  3 files updated, 0 files merged, 0 files removed, 0 files unresolved
> +
> +  $ cat clone/foo/bar/z.txt
> +  z1
> +  z2
> +  z3
> +
> +  $ cat access.log
> +  * "GET /?cmd=capabilities HTTP/1.1" 200 - (glob)
> +  * "GET /?cmd=batch HTTP/1.1" 200 - * (glob)
> +  * "GET /?cmd=getbundle HTTP/1.1" 200 - * (glob)
> +  * "GET /foo?cmd=capabilities HTTP/1.1" 200 - (glob)
> +  * "GET /foo?cmd=batch HTTP/1.1" 200 - * (glob)
> +  * "GET /foo?cmd=getbundle HTTP/1.1" 200 - * (glob)
> +  * "GET /foo/bar?cmd=capabilities HTTP/1.1" 200 - (glob)
> +  * "GET /foo/bar?cmd=batch HTTP/1.1" 200 - * (glob)
> +  * "GET /foo/bar?cmd=getbundle HTTP/1.1" 200 - * (glob)
> +
> +  $ killdaemons.py
> +  $ rm hg1.pid error.log access.log
> +  $ cd repo
> +#endif
> +
> Enable progress extension for archive tests:
> 
>   $ cp $HGRCPATH $HGRCPATH.no-progress

On Sat, 15 Apr 2017 19:00:48 -0400, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison <matt_harbison@yahoo.com>
> # Date 1492293940 14400
> #      Sat Apr 15 18:05:40 2017 -0400
> # Node ID 2804acb18b59dec98abb216d813cc0e3ecec4e14
> # Parent  3fd50d5f9314a96f43eb73480763f224c4d05831
> serve: add support for Mercurial subrepositories

Looks good. Queued, thanks.

> +def addwebdirpath(repo, serverpath, webconf):
> +    webconf[serverpath] = repo.root
> +    repo.ui.debug('adding %s = %s\n' % (serverpath, repo.root))
> +
> +    for r in repo.revs('filelog("path:.hgsub")'):
> +        ctx = repo[r]
> +        for subpath in ctx.substate:
> +            ctx.sub(subpath).addwebdirpath(serverpath, webconf)

[...]

> +    @annotatesubrepoerror
> +    def addwebdirpath(self, serverpath, webconf):
> +        cmdutil.addwebdirpath(self._repo, subrelpath(self), webconf)

'serverpath' isn't used. Perhaps this wouldn't work if addwebdirpath() starts
with non-empty serverpath.

On Sun, 16 Apr 2017 06:20:24 -0400, Yuya Nishihara <yuya@tcha.org> wrote:

> On Sat, 15 Apr 2017 19:00:48 -0400, Matt Harbison wrote:
>> # HG changeset patch
>> # User Matt Harbison <matt_harbison@yahoo.com>
>> # Date 1492293940 14400
>> #      Sat Apr 15 18:05:40 2017 -0400
>> # Node ID 2804acb18b59dec98abb216d813cc0e3ecec4e14
>> # Parent  3fd50d5f9314a96f43eb73480763f224c4d05831
>> serve: add support for Mercurial subrepositories
>
> Looks good. Queued, thanks.
>
>> +def addwebdirpath(repo, serverpath, webconf):
>> +    webconf[serverpath] = repo.root
>> +    repo.ui.debug('adding %s = %s\n' % (serverpath, repo.root))
>> +
>> +    for r in repo.revs('filelog("path:.hgsub")'):
>> +        ctx = repo[r]
>> +        for subpath in ctx.substate:
>> +            ctx.sub(subpath).addwebdirpath(serverpath, webconf)
>
> [...]
>
>> +    @annotatesubrepoerror
>> +    def addwebdirpath(self, serverpath, webconf):
>> +        cmdutil.addwebdirpath(self._repo, subrelpath(self), webconf)
>
> 'serverpath' isn't used. Perhaps this wouldn't work if addwebdirpath()  
> starts
> with non-empty serverpath.

Oops.  It will be a problem with the with the -S + --web-conf patch I  
have, but I'll wait until after the freeze for that.  As it is now,  
there's no way for serverpath to start non-empty.