Submitter | Pierre-Yves David |
---|---|
Date | Aug. 5, 2019, 4:36 p.m. |
Message ID | <909da55a9517693f2911.1565022994@nodosa.octopoid.net> |
Download | mbox | patch |
Permalink | /patch/41156/ |
State | Accepted |
Headers | show |
Comments
On Mon, Aug 5, 2019 at 10:04 AM Pierre-Yves David < pierre-yves.david@ens-lyon.org> wrote: > # HG changeset patch > # User Pierre-Yves David <pierre-yves.david@octobus.net> > # Date 1564439752 -7200 > # Tue Jul 30 00:35:52 2019 +0200 > # Node ID 909da55a9517693f2911793272e03cb7f108cfaf > # Parent 5b8a0d3b7596be28a31faac4ffc499da46c0b62e > # EXP-Topic upgrade-select > # Available At https://bitbucket.org/octobus/mercurial-devel/ > # hg pull https://bitbucket.org/octobus/mercurial-devel/ -r > 909da55a9517 > upgrade: add an argument to control manifest upgrade > Before we land this and the next commit, I'd like you to think about: 1) Filelog support. If we have a flag for manifests and changelogs, shouldn't there be one for files? 2) How the presence of multiple flags interacts. Assuming we add a flag to control files, what happens when you supply --manifest --changelog or --no-manifest --no-changelog? Does that behavior make sense? 3) How do we extend this selection to other storage primitives? Do we need to keep adding arguments for each storage primitive type? I haven't thought this one through, but I have a suspicion that once someone does, we'll decide that having arguments named after actions and taking nouns will be the most scalable solution. e.g. --include manifest --exclude changeset --exclude files. (If you go this route please do not use include/exclude as we may want to reserve those for specifying a fileset of files to upgrade, like many other commands.) > > The argument can be used to only "clone" manifest revlog or clone all of > them > but this one. The selection will make more sense once we have a > `--changelog` > flag in the next changesets. > > diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py > --- a/mercurial/debugcommands.py > +++ b/mercurial/debugcommands.py > @@ -2848,8 +2848,9 @@ def debugupdatecaches(ui, repo, *pats, * > ('o', 'optimize', [], _('extra optimization to perform'), _('NAME')), > ('', 'run', False, _('performs an upgrade')), > ('', 'backup', True, _('keep the old repository content around')), > + ('', 'manifest', None, _('select the manifest for upgrade')), > ]) > -def debugupgraderepo(ui, repo, run=False, optimize=None, backup=True): > +def debugupgraderepo(ui, repo, run=False, optimize=None, backup=True, > **opts): > """upgrade a repository to use different features > > If no arguments are specified, the repository is evaluated for upgrade > @@ -2867,9 +2868,15 @@ def debugupgraderepo(ui, repo, run=False > rename some directories inside the ``.hg`` directory. On most > machines, this > should complete almost instantaneously and the chances of a consumer > being > unable to access the repository should be low. > + > + By default, all revlog will be upgraded. You can restrict this using > flag > + such as `--manifest`: > + > + * `--manifest`: only optimize the manifest > + * `--no-manifest`: optimize all revlog but the manifest > """ > return upgrade.upgraderepo(ui, repo, run=run, optimize=optimize, > - backup=backup) > + backup=backup, **opts) > > @command('debugwalk', cmdutil.walkopts, _('[OPTION]... [FILE]...'), > inferrepo=True) > diff --git a/mercurial/upgrade.py b/mercurial/upgrade.py > --- a/mercurial/upgrade.py > +++ b/mercurial/upgrade.py > @@ -866,13 +866,32 @@ def _upgraderepo(ui, srcrepo, dstrepo, r > > return backuppath > > -def upgraderepo(ui, repo, run=False, optimize=None, backup=True): > +def upgraderepo(ui, repo, run=False, optimize=None, backup=True, > + manifest=None): > """Upgrade a repository in place.""" > if optimize is None: > optimize = [] > optimize = set(legacy_opts_map.get(o, o) for o in optimize) > repo = repo.unfiltered() > > + revlogs = set(UPGRADE_ALL_REVLOGS) > + specentries = (('m', manifest),) > + specified = [(y, x) for (y, x) in specentries if x is not None] > + if specified: > + # we have some limitation on revlogs to be recloned > + if any(x for y, x in specified): > + revlogs = set() > + for r, enabled in specified: > + if enabled: > + if r == 'm': > + revlogs.add(UPGRADE_MANIFEST) > + else: > + # none are enabled > + for r, __ in specified: > + if r == 'm': > + revlogs.discard(UPGRADE_MANIFEST) > + > + > # Ensure the repository can be upgraded. > missingreqs = requiredsourcerequirements(repo) - repo.requirements > if missingreqs: > @@ -1022,7 +1041,7 @@ def upgraderepo(ui, repo, run=False, opt > > with dstrepo.wlock(), dstrepo.lock(): > backuppath = _upgraderepo(ui, repo, dstrepo, newreqs, > - upgradeactions) > + upgradeactions, revlogs=revlogs) > if not (backup or backuppath is None): > ui.write(_('removing old repository content%s\n') % > backuppath) > repo.vfs.rmtree(backuppath, forcibly=True) > diff --git a/tests/test-completion.t b/tests/test-completion.t > --- a/tests/test-completion.t > +++ b/tests/test-completion.t > @@ -312,7 +312,7 @@ Show all commands + options > debuguigetpass: prompt > debuguiprompt: prompt > debugupdatecaches: > - debugupgraderepo: optimize, run, backup > + debugupgraderepo: optimize, run, backup, manifest > debugwalk: include, exclude > debugwhyunstable: > debugwireargs: three, four, five, ssh, remotecmd, insecure > diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t > --- a/tests/test-upgrade-repo.t > +++ b/tests/test-upgrade-repo.t > @@ -518,9 +518,126 @@ unless --no-backup is passed > removing temporary repository $TESTTMP/upgradegd/.hg/upgrade.* (glob) > $ ls -1 .hg/ | grep upgradebackup > [1] > + > +We can restrict optimization to some revlog: > + > + $ hg debugupgrade --optimize re-delta-parent --run --manifest > --no-backup --debug --traceback > + upgrade will perform the following actions: > + > + requirements > + preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, > store > + > + re-delta-parent > + deltas within internal storage will choose a new base revision if > needed > + > + beginning upgrade... > + repository locked and read-only > + creating temporary repository to stage migrated data: > $TESTTMP/upgradegd/.hg/upgrade.* (glob) > + (it is safe to interrupt this process any time before data migration > completes) > + migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in > changelog) > + migrating 917 bytes in store; 401 bytes tracked data > + migrating 3 filelogs containing 3 revisions (192 bytes in store; 0 > bytes tracked data) > + blindly copying data/f0.i containing 1 revisions > + blindly copying data/f1.i containing 1 revisions > + blindly copying data/f2.i containing 1 revisions > + finished migrating 3 filelog revisions across 3 filelogs; change in > size: 0 bytes > + migrating 1 manifests containing 3 revisions (349 bytes in store; 220 > bytes tracked data) > + cloning 3 revisions from 00manifest.i > + finished migrating 3 manifest revisions across 1 manifests; change in > size: 0 bytes > + migrating changelog containing 3 revisions (376 bytes in store; 181 > bytes tracked data) > + blindly copying 00changelog.i containing 3 revisions > + finished migrating 3 changelog revisions; change in size: 0 bytes > + finished migrating 9 total revisions; total change in store size: 0 > bytes > + copying phaseroots > + data fully migrated to temporary repository > + marking source repository as being upgraded; clients will be unable to > read from repository > + starting in-place swap of repository data > + replaced files will be backed up at > $TESTTMP/upgradegd/.hg/upgradebackup.* (glob) > + replacing store... > + store replacement complete; repository was inconsistent for *s (glob) > + finalizing requirements file and making repository readable again > + removing old repository content$TESTTMP/upgradegd/.hg/upgradebackup.* > (glob) > + removing temporary repository $TESTTMP/upgradegd/.hg/upgrade.* (glob) > + > +Check that the repo still works fine > + > + $ hg log -G --patch > + @ changeset: 2:b5a3b78015e5 > + | tag: tip > + | parent: 0:ba592bf28da2 > + | user: test > + | date: Thu Jan 01 00:00:00 1970 +0000 > + | summary: add f2 > + | > + | > + | o changeset: 1:da8c0fc4833c > + |/ user: test > + | date: Thu Jan 01 00:00:00 1970 +0000 > + | summary: add f1 > + | > + | > + o changeset: 0:ba592bf28da2 > + user: test > + date: Thu Jan 01 00:00:00 1970 +0000 > + summary: initial > + > + > + > + $ hg verify > + checking changesets > + checking manifests > + crosschecking files in changesets and manifests > + checking files > + checked 3 changesets with 3 changes to 3 files > + > +Check we can select negatively > + > + $ hg debugupgrade --optimize re-delta-parent --run --no-manifest > --no-backup --debug --traceback > + upgrade will perform the following actions: > + > + requirements > + preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, > store > + > + re-delta-parent > + deltas within internal storage will choose a new base revision if > needed > + > + beginning upgrade... > + repository locked and read-only > + creating temporary repository to stage migrated data: > $TESTTMP/upgradegd/.hg/upgrade.* (glob) > + (it is safe to interrupt this process any time before data migration > completes) > + migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in > changelog) > + migrating 917 bytes in store; 401 bytes tracked data > + migrating 3 filelogs containing 3 revisions (192 bytes in store; 0 > bytes tracked data) > + cloning 1 revisions from data/f0.i > + cloning 1 revisions from data/f1.i > + cloning 1 revisions from data/f2.i > + finished migrating 3 filelog revisions across 3 filelogs; change in > size: 0 bytes > + migrating 1 manifests containing 3 revisions (349 bytes in store; 220 > bytes tracked data) > + blindly copying 00manifest.i containing 3 revisions > + finished migrating 3 manifest revisions across 1 manifests; change in > size: 0 bytes > + migrating changelog containing 3 revisions (376 bytes in store; 181 > bytes tracked data) > + cloning 3 revisions from 00changelog.i > + finished migrating 3 changelog revisions; change in size: 0 bytes > + finished migrating 9 total revisions; total change in store size: 0 > bytes > + copying phaseroots > + data fully migrated to temporary repository > + marking source repository as being upgraded; clients will be unable to > read from repository > + starting in-place swap of repository data > + replaced files will be backed up at > $TESTTMP/upgradegd/.hg/upgradebackup.* (glob) > + replacing store... > + store replacement complete; repository was inconsistent for *s (glob) > + finalizing requirements file and making repository readable again > + removing old repository content$TESTTMP/upgradegd/.hg/upgradebackup.* > (glob) > + removing temporary repository $TESTTMP/upgradegd/.hg/upgrade.* (glob) > + $ hg verify > + checking changesets > + checking manifests > + crosschecking files in changesets and manifests > + checking files > + checked 3 changesets with 3 changes to 3 files > + > $ cd .. > > - > store files with special filenames aren't encoded during copy > > $ hg init store-filenames > _______________________________________________ > Mercurial-devel mailing list > Mercurial-devel@mercurial-scm.org > https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel >
On 8/6/19 4:44 AM, Gregory Szorc wrote: > On Mon, Aug 5, 2019 at 10:04 AM Pierre-Yves David > <pierre-yves.david@ens-lyon.org <mailto:pierre-yves.david@ens-lyon.org>> > wrote: > > # HG changeset patch > # User Pierre-Yves David <pierre-yves.david@octobus.net > <mailto:pierre-yves.david@octobus.net>> > # Date 1564439752 -7200 > # Tue Jul 30 00:35:52 2019 +0200 > # Node ID 909da55a9517693f2911793272e03cb7f108cfaf > # Parent 5b8a0d3b7596be28a31faac4ffc499da46c0b62e > # EXP-Topic upgrade-select > # Available At https://bitbucket.org/octobus/mercurial-devel/ > # hg pull > https://bitbucket.org/octobus/mercurial-devel/ -r 909da55a9517 > upgrade: add an argument to control manifest upgrade > > > Before we land this and the next commit, I'd like you to think about: > > 1) Filelog support. If we have a flag for manifests and changelogs, > shouldn't there be one for files? With the next changeset, we can select all filelog and no-filelog. We could introduce a flag dedicated to filelog, that would match on the filename. I did not had a usecase for it so I did not. > 2) How the presence of multiple flags interacts. Assuming we add a flag > to control files, what happens when you supply --manifest --changelog or > --no-manifest --no-changelog? Does that behavior make sense? Multiple flag combine to allow a finer selection, here is the extract from the help (with the next changesets) * `--manifest`: only optimize the manifest * `--no-manifest`: optimize all revlog but the manifest * `--changelog`: optimize the changelog only * `--no-changelog --no-manifest`: optimize filelogs only > 3) How do we extend this selection to other storage primitives? Do we > need to keep adding arguments for each storage primitive type? Multiple commands in mercurial use the --manifest/--changelog flag. So we are not introducing any new questions with this series.
Patch
diff --git a/mercurial/debugcommands.py b/mercurial/debugcommands.py --- a/mercurial/debugcommands.py +++ b/mercurial/debugcommands.py @@ -2848,8 +2848,9 @@ def debugupdatecaches(ui, repo, *pats, * ('o', 'optimize', [], _('extra optimization to perform'), _('NAME')), ('', 'run', False, _('performs an upgrade')), ('', 'backup', True, _('keep the old repository content around')), + ('', 'manifest', None, _('select the manifest for upgrade')), ]) -def debugupgraderepo(ui, repo, run=False, optimize=None, backup=True): +def debugupgraderepo(ui, repo, run=False, optimize=None, backup=True, **opts): """upgrade a repository to use different features If no arguments are specified, the repository is evaluated for upgrade @@ -2867,9 +2868,15 @@ def debugupgraderepo(ui, repo, run=False rename some directories inside the ``.hg`` directory. On most machines, this should complete almost instantaneously and the chances of a consumer being unable to access the repository should be low. + + By default, all revlog will be upgraded. You can restrict this using flag + such as `--manifest`: + + * `--manifest`: only optimize the manifest + * `--no-manifest`: optimize all revlog but the manifest """ return upgrade.upgraderepo(ui, repo, run=run, optimize=optimize, - backup=backup) + backup=backup, **opts) @command('debugwalk', cmdutil.walkopts, _('[OPTION]... [FILE]...'), inferrepo=True) diff --git a/mercurial/upgrade.py b/mercurial/upgrade.py --- a/mercurial/upgrade.py +++ b/mercurial/upgrade.py @@ -866,13 +866,32 @@ def _upgraderepo(ui, srcrepo, dstrepo, r return backuppath -def upgraderepo(ui, repo, run=False, optimize=None, backup=True): +def upgraderepo(ui, repo, run=False, optimize=None, backup=True, + manifest=None): """Upgrade a repository in place.""" if optimize is None: optimize = [] optimize = set(legacy_opts_map.get(o, o) for o in optimize) repo = repo.unfiltered() + revlogs = set(UPGRADE_ALL_REVLOGS) + specentries = (('m', manifest),) + specified = [(y, x) for (y, x) in specentries if x is not None] + if specified: + # we have some limitation on revlogs to be recloned + if any(x for y, x in specified): + revlogs = set() + for r, enabled in specified: + if enabled: + if r == 'm': + revlogs.add(UPGRADE_MANIFEST) + else: + # none are enabled + for r, __ in specified: + if r == 'm': + revlogs.discard(UPGRADE_MANIFEST) + + # Ensure the repository can be upgraded. missingreqs = requiredsourcerequirements(repo) - repo.requirements if missingreqs: @@ -1022,7 +1041,7 @@ def upgraderepo(ui, repo, run=False, opt with dstrepo.wlock(), dstrepo.lock(): backuppath = _upgraderepo(ui, repo, dstrepo, newreqs, - upgradeactions) + upgradeactions, revlogs=revlogs) if not (backup or backuppath is None): ui.write(_('removing old repository content%s\n') % backuppath) repo.vfs.rmtree(backuppath, forcibly=True) diff --git a/tests/test-completion.t b/tests/test-completion.t --- a/tests/test-completion.t +++ b/tests/test-completion.t @@ -312,7 +312,7 @@ Show all commands + options debuguigetpass: prompt debuguiprompt: prompt debugupdatecaches: - debugupgraderepo: optimize, run, backup + debugupgraderepo: optimize, run, backup, manifest debugwalk: include, exclude debugwhyunstable: debugwireargs: three, four, five, ssh, remotecmd, insecure diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t --- a/tests/test-upgrade-repo.t +++ b/tests/test-upgrade-repo.t @@ -518,9 +518,126 @@ unless --no-backup is passed removing temporary repository $TESTTMP/upgradegd/.hg/upgrade.* (glob) $ ls -1 .hg/ | grep upgradebackup [1] + +We can restrict optimization to some revlog: + + $ hg debugupgrade --optimize re-delta-parent --run --manifest --no-backup --debug --traceback + upgrade will perform the following actions: + + requirements + preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store + + re-delta-parent + deltas within internal storage will choose a new base revision if needed + + beginning upgrade... + repository locked and read-only + creating temporary repository to stage migrated data: $TESTTMP/upgradegd/.hg/upgrade.* (glob) + (it is safe to interrupt this process any time before data migration completes) + migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in changelog) + migrating 917 bytes in store; 401 bytes tracked data + migrating 3 filelogs containing 3 revisions (192 bytes in store; 0 bytes tracked data) + blindly copying data/f0.i containing 1 revisions + blindly copying data/f1.i containing 1 revisions + blindly copying data/f2.i containing 1 revisions + finished migrating 3 filelog revisions across 3 filelogs; change in size: 0 bytes + migrating 1 manifests containing 3 revisions (349 bytes in store; 220 bytes tracked data) + cloning 3 revisions from 00manifest.i + finished migrating 3 manifest revisions across 1 manifests; change in size: 0 bytes + migrating changelog containing 3 revisions (376 bytes in store; 181 bytes tracked data) + blindly copying 00changelog.i containing 3 revisions + finished migrating 3 changelog revisions; change in size: 0 bytes + finished migrating 9 total revisions; total change in store size: 0 bytes + copying phaseroots + data fully migrated to temporary repository + marking source repository as being upgraded; clients will be unable to read from repository + starting in-place swap of repository data + replaced files will be backed up at $TESTTMP/upgradegd/.hg/upgradebackup.* (glob) + replacing store... + store replacement complete; repository was inconsistent for *s (glob) + finalizing requirements file and making repository readable again + removing old repository content$TESTTMP/upgradegd/.hg/upgradebackup.* (glob) + removing temporary repository $TESTTMP/upgradegd/.hg/upgrade.* (glob) + +Check that the repo still works fine + + $ hg log -G --patch + @ changeset: 2:b5a3b78015e5 + | tag: tip + | parent: 0:ba592bf28da2 + | user: test + | date: Thu Jan 01 00:00:00 1970 +0000 + | summary: add f2 + | + | + | o changeset: 1:da8c0fc4833c + |/ user: test + | date: Thu Jan 01 00:00:00 1970 +0000 + | summary: add f1 + | + | + o changeset: 0:ba592bf28da2 + user: test + date: Thu Jan 01 00:00:00 1970 +0000 + summary: initial + + + + $ hg verify + checking changesets + checking manifests + crosschecking files in changesets and manifests + checking files + checked 3 changesets with 3 changes to 3 files + +Check we can select negatively + + $ hg debugupgrade --optimize re-delta-parent --run --no-manifest --no-backup --debug --traceback + upgrade will perform the following actions: + + requirements + preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store + + re-delta-parent + deltas within internal storage will choose a new base revision if needed + + beginning upgrade... + repository locked and read-only + creating temporary repository to stage migrated data: $TESTTMP/upgradegd/.hg/upgrade.* (glob) + (it is safe to interrupt this process any time before data migration completes) + migrating 9 total revisions (3 in filelogs, 3 in manifests, 3 in changelog) + migrating 917 bytes in store; 401 bytes tracked data + migrating 3 filelogs containing 3 revisions (192 bytes in store; 0 bytes tracked data) + cloning 1 revisions from data/f0.i + cloning 1 revisions from data/f1.i + cloning 1 revisions from data/f2.i + finished migrating 3 filelog revisions across 3 filelogs; change in size: 0 bytes + migrating 1 manifests containing 3 revisions (349 bytes in store; 220 bytes tracked data) + blindly copying 00manifest.i containing 3 revisions + finished migrating 3 manifest revisions across 1 manifests; change in size: 0 bytes + migrating changelog containing 3 revisions (376 bytes in store; 181 bytes tracked data) + cloning 3 revisions from 00changelog.i + finished migrating 3 changelog revisions; change in size: 0 bytes + finished migrating 9 total revisions; total change in store size: 0 bytes + copying phaseroots + data fully migrated to temporary repository + marking source repository as being upgraded; clients will be unable to read from repository + starting in-place swap of repository data + replaced files will be backed up at $TESTTMP/upgradegd/.hg/upgradebackup.* (glob) + replacing store... + store replacement complete; repository was inconsistent for *s (glob) + finalizing requirements file and making repository readable again + removing old repository content$TESTTMP/upgradegd/.hg/upgradebackup.* (glob) + removing temporary repository $TESTTMP/upgradegd/.hg/upgrade.* (glob) + $ hg verify + checking changesets + checking manifests + crosschecking files in changesets and manifests + checking files + checked 3 changesets with 3 changes to 3 files + $ cd .. - store files with special filenames aren't encoded during copy $ hg init store-filenames