Patchwork [6,of,8] upgrade: add an argument to control manifest upgrade

# HG changeset patch
# User Pierre-Yves David <pierre-yves.david@octobus.net>
# Date 1564439752 -7200
#      Tue Jul 30 00:35:52 2019 +0200
# Node ID 909da55a9517693f2911793272e03cb7f108cfaf
# Parent  5b8a0d3b7596be28a31faac4ffc499da46c0b62e
# EXP-Topic upgrade-select
# Available At https://bitbucket.org/octobus/mercurial-devel/
#              hg pull https://bitbucket.org/octobus/mercurial-devel/ -r 909da55a9517
upgrade: add an argument to control manifest upgrade

The argument can be used to only "clone" manifest revlog or clone all of them
but this one. The selection will make more sense once we have a `--changelog`
flag in the next changesets.

On Mon, Aug 5, 2019 at 10:04 AM Pierre-Yves David <
pierre-yves.david@ens-lyon.org> wrote:

> # HG changeset patch
> # User Pierre-Yves David <pierre-yves.david@octobus.net>
> # Date 1564439752 -7200
> #      Tue Jul 30 00:35:52 2019 +0200
> # Node ID 909da55a9517693f2911793272e03cb7f108cfaf
> # Parent  5b8a0d3b7596be28a31faac4ffc499da46c0b62e
> # EXP-Topic upgrade-select
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #              hg pull https://bitbucket.org/octobus/mercurial-devel/ -r
> 909da55a9517
> upgrade: add an argument to control manifest upgrade
>

Before we land this and the next commit, I'd like you to think about:

1) Filelog support. If we have a flag for manifests and changelogs,
shouldn't there be one for files?
2) How the presence of multiple flags interacts. Assuming we add a flag to
control files, what happens when you supply --manifest --changelog or
--no-manifest --no-changelog? Does that behavior make sense?
3) How do we extend this selection to other storage primitives? Do we need
to keep adding arguments for each storage primitive type?

I haven't thought this one through, but I have a suspicion that once
someone does, we'll decide that having arguments named after actions and
taking nouns will be the most scalable solution. e.g. --include manifest
--exclude changeset --exclude files. (If you go this route please do not
use include/exclude as we may want to reserve those for specifying a
fileset of files to upgrade, like many other commands.)


>
> The argument can be used to only "clone" manifest revlog or clone all of
> them
> but this one. The selection will make more sense once we have a
> `--changelog`
> flag in the next changesets.
>
> diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py
> --- a/mercurial/debugcommands.py
> +++ b/mercurial/debugcommands.py
> @@ -2848,8 +2848,9 @@ def debugupdatecaches(ui, repo, *pats, *
>      ('o', 'optimize', [], _('extra optimization to perform'), _('NAME')),
>      ('', 'run', False, _('performs an upgrade')),
>      ('', 'backup', True, _('keep the old repository content around')),
> +    ('', 'manifest', None, _('select the manifest for upgrade')),
>  ])
> -def debugupgraderepo(ui, repo, run=False, optimize=None, backup=True):
> +def debugupgraderepo(ui, repo, run=False, optimize=None, backup=True,
> **opts):
>      """upgrade a repository to use different features
>
>      If no arguments are specified, the repository is evaluated for upgrade
> @@ -2867,9 +2868,15 @@ def debugupgraderepo(ui, repo, run=False
>      rename some directories inside the ``.hg`` directory. On most
> machines, this
>      should complete almost instantaneously and the chances of a consumer
> being
>      unable to access the repository should be low.
> +
> +    By default, all revlog will be upgraded. You can restrict this using
> flag
> +    such as `--manifest`:
> +
> +      * `--manifest`: only optimize the manifest
> +      * `--no-manifest`: optimize all revlog but the manifest
>      """
>      return upgrade.upgraderepo(ui, repo, run=run, optimize=optimize,
> -                               backup=backup)
> +                               backup=backup, **opts)
>
>  @command('debugwalk', cmdutil.walkopts, _('[OPTION]... [FILE]...'),
>           inferrepo=True)
> diff --git a/mercurial/upgrade.py b/mercurial/upgrade.py
> --- a/mercurial/upgrade.py
> +++ b/mercurial/upgrade.py
> @@ -866,13 +866,32 @@ def _upgraderepo(ui, srcrepo, dstrepo, r
>
>      return backuppath
>
> -def upgraderepo(ui, repo, run=False, optimize=None, backup=True):
> +def upgraderepo(ui, repo, run=False, optimize=None, backup=True,
> +                manifest=None):
>      """Upgrade a repository in place."""
>      if optimize is None:
>          optimize = []
>      optimize = set(legacy_opts_map.get(o, o) for o in optimize)
>      repo = repo.unfiltered()
>
> +    revlogs = set(UPGRADE_ALL_REVLOGS)
> +    specentries = (('m', manifest),)
> +    specified = [(y, x) for (y, x) in specentries if x is not None]
> +    if specified:
> +        # we have some limitation on revlogs to be recloned
> +        if any(x for y, x in specified):
> +            revlogs = set()
> +            for r, enabled in specified:
> +                if enabled:
> +                    if r == 'm':
> +                        revlogs.add(UPGRADE_MANIFEST)
> +        else:
> +            # none are enabled
> +            for r, __ in specified:
> +                if r == 'm':
> +                    revlogs.discard(UPGRADE_MANIFEST)
> +
> +
>      # Ensure the repository can be upgraded.
>      missingreqs = requiredsourcerequirements(repo) - repo.requirements
>      if missingreqs:
> @@ -1022,7 +1041,7 @@ def upgraderepo(ui, repo, run=False, opt
>
>              with dstrepo.wlock(), dstrepo.lock():
>                  backuppath = _upgraderepo(ui, repo, dstrepo, newreqs,
> -                                          upgradeactions)
> +                                          upgradeactions, revlogs=revlogs)
>              if not (backup or backuppath is None):
>                  ui.write(_('removing old repository content%s\n') %
> backuppath)
>                  repo.vfs.rmtree(backuppath, forcibly=True)
> diff --git a/tests/test-completion.t b/tests/test-completion.t
> --- a/tests/test-completion.t
> +++ b/tests/test-completion.t
> @@ -312,7 +312,7 @@ Show all commands + options
>    debuguigetpass: prompt
>    debuguiprompt: prompt
>    debugupdatecaches:
> -  debugupgraderepo: optimize, run, backup
> +  debugupgraderepo: optimize, run, backup, manifest
>    debugwalk: include, exclude
>    debugwhyunstable:
>    debugwireargs: three, four, five, ssh, remotecmd, insecure
> diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t
> --- a/tests/test-upgrade-repo.t
> +++ b/tests/test-upgrade-repo.t
> @@ -518,9 +518,126 @@ unless --no-backup is passed
>    removing temporary repository $TESTTMP/upgradegd/.hg/upgrade.* (glob)
>    $ ls -1 .hg/ | grep upgradebackup
>    [1]
> +
> +We can restrict optimization to some revlog:
> +
> +  $ hg debugupgrade --optimize re-delta-parent --run --manifest
> --no-backup --debug --traceback
> +  upgrade will perform the following actions:
> +
> +  requirements
> +     preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog,
> store
> +
> +  re-delta-parent
> +     deltas within internal storage will choose a new base revision if
> needed
> +
> +  beginning upgrade...
> +  repository locked and read-only
> +  creating temporary repository to stage migrated data:
> $TESTTMP/upgradegd/.hg/upgrade.* (glob)
> +  (it is safe to interrupt this process any time before data migration
> completes)
> +  migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in
> changelog)
> +  migrating 917 bytes in store; 401 bytes tracked data
> +  migrating 3 filelogs containing 3 revisions (192 bytes in store; 0
> bytes tracked data)
> +  blindly copying data/f0.i containing 1 revisions
> +  blindly copying data/f1.i containing 1 revisions
> +  blindly copying data/f2.i containing 1 revisions
> +  finished migrating 3 filelog revisions across 3 filelogs; change in
> size: 0 bytes
> +  migrating 1 manifests containing 3 revisions (349 bytes in store; 220
> bytes tracked data)
> +  cloning 3 revisions from 00manifest.i
> +  finished migrating 3 manifest revisions across 1 manifests; change in
> size: 0 bytes
> +  migrating changelog containing 3 revisions (376 bytes in store; 181
> bytes tracked data)
> +  blindly copying 00changelog.i containing 3 revisions
> +  finished migrating 3 changelog revisions; change in size: 0 bytes
> +  finished migrating 9 total revisions; total change in store size: 0
> bytes
> +  copying phaseroots
> +  data fully migrated to temporary repository
> +  marking source repository as being upgraded; clients will be unable to
> read from repository
> +  starting in-place swap of repository data
> +  replaced files will be backed up at
> $TESTTMP/upgradegd/.hg/upgradebackup.* (glob)
> +  replacing store...
> +  store replacement complete; repository was inconsistent for *s (glob)
> +  finalizing requirements file and making repository readable again
> +  removing old repository content$TESTTMP/upgradegd/.hg/upgradebackup.*
> (glob)
> +  removing temporary repository $TESTTMP/upgradegd/.hg/upgrade.* (glob)
> +
> +Check that the repo still works fine
> +
> +  $ hg log -G --patch
> +  @  changeset:   2:b5a3b78015e5
> +  |  tag:         tip
> +  |  parent:      0:ba592bf28da2
> +  |  user:        test
> +  |  date:        Thu Jan 01 00:00:00 1970 +0000
> +  |  summary:     add f2
> +  |
> +  |
> +  | o  changeset:   1:da8c0fc4833c
> +  |/   user:        test
> +  |    date:        Thu Jan 01 00:00:00 1970 +0000
> +  |    summary:     add f1
> +  |
> +  |
> +  o  changeset:   0:ba592bf28da2
> +     user:        test
> +     date:        Thu Jan 01 00:00:00 1970 +0000
> +     summary:     initial
> +
> +
> +
> +  $ hg verify
> +  checking changesets
> +  checking manifests
> +  crosschecking files in changesets and manifests
> +  checking files
> +  checked 3 changesets with 3 changes to 3 files
> +
> +Check we can select negatively
> +
> +  $ hg debugupgrade --optimize re-delta-parent --run --no-manifest
> --no-backup --debug --traceback
> +  upgrade will perform the following actions:
> +
> +  requirements
> +     preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog,
> store
> +
> +  re-delta-parent
> +     deltas within internal storage will choose a new base revision if
> needed
> +
> +  beginning upgrade...
> +  repository locked and read-only
> +  creating temporary repository to stage migrated data:
> $TESTTMP/upgradegd/.hg/upgrade.* (glob)
> +  (it is safe to interrupt this process any time before data migration
> completes)
> +  migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in
> changelog)
> +  migrating 917 bytes in store; 401 bytes tracked data
> +  migrating 3 filelogs containing 3 revisions (192 bytes in store; 0
> bytes tracked data)
> +  cloning 1 revisions from data/f0.i
> +  cloning 1 revisions from data/f1.i
> +  cloning 1 revisions from data/f2.i
> +  finished migrating 3 filelog revisions across 3 filelogs; change in
> size: 0 bytes
> +  migrating 1 manifests containing 3 revisions (349 bytes in store; 220
> bytes tracked data)
> +  blindly copying 00manifest.i containing 3 revisions
> +  finished migrating 3 manifest revisions across 1 manifests; change in
> size: 0 bytes
> +  migrating changelog containing 3 revisions (376 bytes in store; 181
> bytes tracked data)
> +  cloning 3 revisions from 00changelog.i
> +  finished migrating 3 changelog revisions; change in size: 0 bytes
> +  finished migrating 9 total revisions; total change in store size: 0
> bytes
> +  copying phaseroots
> +  data fully migrated to temporary repository
> +  marking source repository as being upgraded; clients will be unable to
> read from repository
> +  starting in-place swap of repository data
> +  replaced files will be backed up at
> $TESTTMP/upgradegd/.hg/upgradebackup.* (glob)
> +  replacing store...
> +  store replacement complete; repository was inconsistent for *s (glob)
> +  finalizing requirements file and making repository readable again
> +  removing old repository content$TESTTMP/upgradegd/.hg/upgradebackup.*
> (glob)
> +  removing temporary repository $TESTTMP/upgradegd/.hg/upgrade.* (glob)
> +  $ hg verify
> +  checking changesets
> +  checking manifests
> +  crosschecking files in changesets and manifests
> +  checking files
> +  checked 3 changesets with 3 changes to 3 files
> +
>    $ cd ..
>
> -
>  store files with special filenames aren't encoded during copy
>
>    $ hg init store-filenames
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@mercurial-scm.org
> https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel
>

On 8/6/19 4:44 AM, Gregory Szorc wrote:
> On Mon, Aug 5, 2019 at 10:04 AM Pierre-Yves David 
> <pierre-yves.david@ens-lyon.org <mailto:pierre-yves.david@ens-lyon.org>> 
> wrote:
> 
>     # HG changeset patch
>     # User Pierre-Yves David <pierre-yves.david@octobus.net
>     <mailto:pierre-yves.david@octobus.net>>
>     # Date 1564439752 -7200
>     #      Tue Jul 30 00:35:52 2019 +0200
>     # Node ID 909da55a9517693f2911793272e03cb7f108cfaf
>     # Parent  5b8a0d3b7596be28a31faac4ffc499da46c0b62e
>     # EXP-Topic upgrade-select
>     # Available At https://bitbucket.org/octobus/mercurial-devel/
>     #              hg pull
>     https://bitbucket.org/octobus/mercurial-devel/ -r 909da55a9517
>     upgrade: add an argument to control manifest upgrade
> 
> 
> Before we land this and the next commit, I'd like you to think about:
> 
> 1) Filelog support. If we have a flag for manifests and changelogs, 
> shouldn't there be one for files?

With the next changeset, we can select all filelog and no-filelog. We 
could introduce a flag dedicated to filelog, that would match on the 
filename. I did not had a usecase for it so I did not.

> 2) How the presence of multiple flags interacts. Assuming we add a flag 
> to control files, what happens when you supply --manifest --changelog or 
> --no-manifest --no-changelog? Does that behavior make sense?

Multiple flag combine to allow a finer selection, here is the extract 
from the help (with the next changesets)

       * `--manifest`: only optimize the manifest
       * `--no-manifest`: optimize all revlog but the manifest
       * `--changelog`: optimize the changelog only
       * `--no-changelog --no-manifest`: optimize filelogs only

> 3) How do we extend this selection to other storage primitives? Do we 
> need to keep adding arguments for each storage primitive type?

Multiple commands in mercurial use the --manifest/--changelog flag. So 
we are not introducing any new questions with this series.