Patchwork D8431: upgrade: support upgrade and downgrade from persistent nodemap

login
register
mail settings
Submitter phabricator
Date April 15, 2020, 8:17 p.m.
Message ID <differential-rev-PHID-DREV-6ziegtts2vdzdwgg6ice-req@mercurial-scm.org>
Download mbox | patch
Permalink /patch/46135/
State Superseded
Headers show

Comments

phabricator - April 15, 2020, 8:17 p.m.
marmoute created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  The requirements is not recognised and dealt with and the associated files
  properly handled.
  
  The persistent nodemap should be ready for usage in the field now.

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D8431

AFFECTED FILES
  mercurial/upgrade.py
  tests/test-persistent-nodemap.t

CHANGE DETAILS




To: marmoute, #hg-reviewers
Cc: mercurial-devel
phabricator - April 16, 2020, 1:51 p.m.
mharbison72 added a comment.
mharbison72 accepted this revision.


  s/requirements is not/requirement is now/ in the summary, but otherwise LGTM

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8431/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8431

To: marmoute, #hg-reviewers, mharbison72
Cc: mharbison72, mercurial-devel
phabricator - May 7, 2020, 1:26 p.m.
Alphare added a comment.
Alphare accepted this revision.


  Same remark as Matt.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8431/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8431

To: marmoute, #hg-reviewers, mharbison72, Alphare
Cc: Alphare, mercurial-patches, mharbison72, mercurial-devel
phabricator - May 7, 2020, 2:13 p.m.
marmoute added a comment.


  In D8431#125976 <https://phab.mercurial-scm.org/D8431#125976>, @mharbison72 wrote:
  
  > s/requirements is not/requirement is now/ in the summary, but otherwise LGTM
  
  Good catch, fixing it now.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8431/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8431

To: marmoute, #hg-reviewers, mharbison72, Alphare
Cc: Alphare, mercurial-patches, mharbison72, mercurial-devel
phabricator - May 7, 2020, 3:17 p.m.
This revision now requires changes to proceed.
durin42 added a comment.
durin42 requested changes to this revision.


  This requirement needs documentation in `mercurial/helptext/internals/requirements.txt`. I've taken the rest of the stack, but am blocking this patch on that change.

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8431/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8431

To: marmoute, #hg-reviewers, mharbison72, Alphare, durin42
Cc: durin42, Alphare, mercurial-patches, mharbison72, mercurial-devel
phabricator - May 7, 2020, 9:59 p.m.
marmoute added a comment.


  Good catch. I updated the internal doc and https://www.mercurial-scm.org/wiki/MissingRequirement

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8431/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8431

To: marmoute, #hg-reviewers, mharbison72, Alphare, durin42
Cc: durin42, Alphare, mercurial-patches, mharbison72, mercurial-devel
phabricator - May 11, 2020, 9:08 p.m.
durin42 added inline comments.

INLINE COMMENTS

> requirements.txt:150
> +The `nodemap` index (mapping nodeid to local revision number) is persisted on
> +disk. This provides speed benefit (if the associated native code is used. The
> +persistent nodemap is only used for two revlogs: the changelog and the manifest

The parenthetical on this line isn't closed. Should we just drop the opening paren?

> requirements.txt:155
> +Support for this requirement was added in Mercurial 5.5 (released August 2020).
> +Note that as of 5.5, only instalation compiled with the Rust extension will
> +benefit from a speedup. The other instalation will do the necessary work to keep

nit: installations (should be plural, also has typo)

> requirements.txt:156
> +Note that as of 5.5, only instalation compiled with the Rust extension will
> +benefit from a speedup. The other instalation will do the necessary work to keep
> +the index up to date, but will suffer a slowdown.

"Other installations ..."

REPOSITORY
  rHG Mercurial

CHANGES SINCE LAST ACTION
  https://phab.mercurial-scm.org/D8431/new/

REVISION DETAIL
  https://phab.mercurial-scm.org/D8431

To: marmoute, #hg-reviewers, mharbison72, Alphare, durin42
Cc: durin42, Alphare, mercurial-patches, mharbison72, mercurial-devel

Patch

diff --git a/tests/test-persistent-nodemap.t b/tests/test-persistent-nodemap.t
--- a/tests/test-persistent-nodemap.t
+++ b/tests/test-persistent-nodemap.t
@@ -439,3 +439,99 @@ 
   .hg/store/00changelog-????????????????.nd: size=121536, sha256=bb414468d225cf52d69132e1237afba34d4346ee2eb81b505027e6197b107f03 (glob) (pure !)
   .hg/store/00changelog-????????????????.nd: size=121536, sha256=909ac727bc4d1c0fda5f7bff3c620c98bd4a2967c143405a1503439e33b377da (glob) (rust !)
   .hg/store/00changelog-????????????????.nd: size=121088, sha256=342d36d30d86dde67d3cb6c002606c4a75bcad665595d941493845066d9c8ee0 (glob) (no-pure no-rust !)
+
+Test upgrade / downgrade
+========================
+
+downgrading
+
+  $ cat << EOF >> .hg/hgrc
+  > [format]
+  > use-persistent-nodemap=no
+  > EOF
+  $ hg debugformat -v
+  format-variant     repo config default
+  fncache:            yes    yes     yes
+  dotencode:          yes    yes     yes
+  generaldelta:       yes    yes     yes
+  sparserevlog:       yes    yes     yes
+  sidedata:            no     no      no
+  persistent-nodemap: yes     no      no
+  copies-sdc:          no     no      no
+  plain-cl-delta:     yes    yes     yes
+  compression:        zlib   zlib    zlib
+  compression-level:  default default default
+  $ hg debugupgraderepo --run --no-backup --quiet
+  upgrade will perform the following actions:
+  
+  requirements
+     preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store
+     removed: persistent-nodemap
+  
+  $ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
+  [1]
+  $ hg debugnodemap --metadata
+
+
+upgrading
+
+  $ cat << EOF >> .hg/hgrc
+  > [format]
+  > use-persistent-nodemap=yes
+  > EOF
+  $ hg debugformat -v
+  format-variant     repo config default
+  fncache:            yes    yes     yes
+  dotencode:          yes    yes     yes
+  generaldelta:       yes    yes     yes
+  sparserevlog:       yes    yes     yes
+  sidedata:            no     no      no
+  persistent-nodemap:  no    yes      no
+  copies-sdc:          no     no      no
+  plain-cl-delta:     yes    yes     yes
+  compression:        zlib   zlib    zlib
+  compression-level:  default default default
+  $ hg debugupgraderepo --run --no-backup --quiet
+  upgrade will perform the following actions:
+  
+  requirements
+     preserved: dotencode, fncache, generaldelta, revlogv1, sparserevlog, store
+     added: persistent-nodemap
+  
+  $ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
+  00changelog-*.nd (glob)
+  00changelog.n
+  00manifest-*.nd (glob)
+  00manifest.n
+
+  $ hg debugnodemap --metadata
+  uid: * (glob)
+  tip-rev: 5005
+  tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
+  data-length: 121088
+  data-unused: 0
+  data-unused: 0.000%
+
+Running unrelated upgrade
+
+  $ hg debugupgraderepo --run --no-backup --quiet --optimize re-delta-all
+  upgrade will perform the following actions:
+  
+  requirements
+     preserved: dotencode, fncache, generaldelta, persistent-nodemap, revlogv1, sparserevlog, store
+  
+  optimisations: re-delta-all
+  
+  $ ls -1 .hg/store/ | egrep '00(changelog|manifest)(\.n|-.*\.nd)'
+  00changelog-*.nd (glob)
+  00changelog.n
+  00manifest-*.nd (glob)
+  00manifest.n
+
+  $ hg debugnodemap --metadata
+  uid: * (glob)
+  tip-rev: 5005
+  tip-node: 90d5d3ba2fc47db50f712570487cb261a68c8ffe
+  data-length: 121088
+  data-unused: 0
+  data-unused: 0.000%
diff --git a/mercurial/upgrade.py b/mercurial/upgrade.py
--- a/mercurial/upgrade.py
+++ b/mercurial/upgrade.py
@@ -78,6 +78,7 @@ 
         localrepo.SPARSEREVLOG_REQUIREMENT,
         localrepo.SIDEDATA_REQUIREMENT,
         localrepo.COPIESSDC_REQUIREMENT,
+        localrepo.NODEMAP_REQUIREMENT,
     }
     for name in compression.compengines:
         engine = compression.compengines[name]
@@ -105,6 +106,7 @@ 
         localrepo.SPARSEREVLOG_REQUIREMENT,
         localrepo.SIDEDATA_REQUIREMENT,
         localrepo.COPIESSDC_REQUIREMENT,
+        localrepo.NODEMAP_REQUIREMENT,
     }
     for name in compression.compengines:
         engine = compression.compengines[name]
@@ -132,6 +134,7 @@ 
         localrepo.SPARSEREVLOG_REQUIREMENT,
         localrepo.SIDEDATA_REQUIREMENT,
         localrepo.COPIESSDC_REQUIREMENT,
+        localrepo.NODEMAP_REQUIREMENT,
     }
     for name in compression.compengines:
         engine = compression.compengines[name]
@@ -990,7 +993,7 @@ 
     Function should return ``True`` if the file is to be copied.
     """
     # Skip revlogs.
-    if path.endswith((b'.i', b'.d')):
+    if path.endswith((b'.i', b'.d', b'.n', b'.nd')):
         return False
     # Skip transaction related files.
     if path.startswith(b'undo'):