Patchwork D6416: changelog: optionally store added and removed files in changeset extras

login
register
mail settings
Submitter phabricator
Date May 22, 2019, 12:33 a.m.
Message ID <differential-rev-PHID-DREV-yxmsx63crkr3chw6af2j-req@phab.mercurial-scm.org>
Download mbox | patch
Permalink /patch/40178/
State Superseded
Headers show

Comments

phabricator - May 22, 2019, 12:33 a.m.
martinvonz created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  As mentioned in an earlier patch, copies._chain() is used a lot in the
  changeset-centric version of pathcopies(). It is expensive because it
  needs to look at the manifest in order to filter out copies whose
  target file has since been removed. I want to store the sets of added
  and removed files in the changeset in order to speed that up. This
  patch does the writing part of that. It could easily be a separate
  config, but it's currently tied to experimental.copies.write-to since
  that's the only real use case (it will also make the {file_*} template
  keywords faster, but I doubt that anyone cares enough about those to
  write extra metadata for them).
  
  The new information is stored in the changeset extras. Since they're
  always subsets of the changeset's "files" list, they're stored as
  indexes into that list. I've stored the indexes as stringified ints
  separated by NUL bytes. The size of 00changelog.d for the hg repo
  increased in size by 0.28% percent (compared to the size with only
  copy information in the changesets, which in turn is 0.17% larger than
  without copy information). We could store only the delta between the
  indexes and we could store them in binary, but the chosen format is
  more readable.
  
  We could also have implemented this as a cache outside the
  changelog. One advantage of doing it that way is that we would get the
  speedups from the {file_*} template keywords also on old
  repos. Another advantage is that it we can rewrite the cache if we
  find a bug in how we calculate the set of files. A disadvantage is
  that it would be more complex. Another is that it would surely use
  more space. We already write the copy information to the changeset
  extras, so it seems like a small step to also write these file sets.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D6416

AFFECTED FILES
  mercurial/changelog.py
  mercurial/localrepo.py
  tests/test-copies-in-changeset.t
  tests/test-copies.t

CHANGE DETAILS




To: martinvonz, #hg-reviewers
Cc: mercurial-devel

Patch

diff --git a/tests/test-copies.t b/tests/test-copies.t
--- a/tests/test-copies.t
+++ b/tests/test-copies.t
@@ -551,14 +551,15 @@ 
 
   $ hg up 2 -q
   $ hg graft -r 4 --base 3 --hidden
-  grafting 4:af28412ec03c "added d, modified b" (tip)
+  grafting 4:af28412ec03c "added d, modified b" (tip) (no-changeset !)
+  grafting 4:6325ca0b7a1c "added d, modified b" (tip) (changeset !)
   merging b1 and b to b1
 
   $ hg l -l1 -p
   @  5 added d, modified b
   |  b1
   ~  diff -r 5a4825cc2926 -r 94a2f1a0e8e2 b1 (no-changeset !)
-  ~  diff -r f5474f5023a8 -r ef7c02d69f3d b1 (changeset !)
+  ~  diff -r df722b7fe2d5 -r ba3ddbbdfd96 b1 (changeset !)
      --- a/b1	Thu Jan 01 00:00:00 1970 +0000
      +++ b/b1	Thu Jan 01 00:00:00 1970 +0000
      @@ -1,1 +1,2 @@
@@ -618,7 +619,8 @@ 
      a
 
   $ hg rebase -r . -d 2 -t :other
-  rebasing 5:5018b1509e94 "added willconflict and d" (tip)
+  rebasing 5:5018b1509e94 "added willconflict and d" (tip) (no-changeset !)
+  rebasing 5:619047c26bf8 "added willconflict and d" (tip) (changeset !)
 
   $ hg up 3 -q
   $ hg l --hidden
@@ -641,4 +643,5 @@ 
 neither of the merging csets will be a descendant of the base revision:
 
   $ hg graft -r 6 --base 4 --hidden -t :other
-  grafting 6:99802e4f1e46 "added willconflict and d" (tip)
+  grafting 6:99802e4f1e46 "added willconflict and d" (tip) (no-changeset !)
+  grafting 6:9ddc6fb3b691 "added willconflict and d" (tip) (changeset !)
diff --git a/tests/test-copies-in-changeset.t b/tests/test-copies-in-changeset.t
--- a/tests/test-copies-in-changeset.t
+++ b/tests/test-copies-in-changeset.t
@@ -5,6 +5,7 @@ 
   > copies.read-from=changeset-only
   > [alias]
   > changesetcopies = log -r . -T 'files: {files}
+  >   {extras % "{ifcontains("files", key, "{key}: {value}\n")}"}
   >   {extras % "{ifcontains("copies", key, "{key}: {value}\n")}"}'
   > showcopies = log -r . -T '{file_copies % "{source} -> {name}\n"}'
   > [extensions]
@@ -24,6 +25,8 @@ 
   $ hg ci -m 'copy a to b, c, and d'
   $ hg changesetcopies
   files: b c d
+  filesadded: 0\x001\x002 (esc)
+  
   p1copies: b\x00a (esc)
   c\x00a (esc)
   d\x00a (esc)
@@ -43,6 +46,9 @@ 
   $ hg ci -m 'rename b to b2'
   $ hg changesetcopies
   files: b b2
+  filesadded: 1
+  filesremoved: 0
+  
   p1copies: b2\x00b (esc)
   $ hg showcopies
   b -> b2
@@ -60,6 +66,7 @@ 
   $ hg ci -m 'move b onto d'
   $ hg changesetcopies
   files: c
+  
   p1copies: c\x00b2 (esc)
   $ hg showcopies
   b2 -> c
@@ -88,6 +95,8 @@ 
   $ hg ci -m 'merge'
   $ hg changesetcopies
   files: g h i
+  filesadded: 0\x001\x002 (esc)
+  
   p1copies: g\x00a (esc)
   i\x00f (esc)
   p2copies: h\x00d (esc)
@@ -102,6 +111,9 @@ 
   $ hg ci -m 'copy a to j' --config experimental.copies.write-to=compatibility
   $ hg changesetcopies
   files: j
+  filesadded: 0
+  filesremoved: 
+  
   p1copies: j\x00a (esc)
   p2copies: 
   $ hg debugdata j 0
@@ -122,6 +134,9 @@ 
   $ hg ci -m 'modify j' --config experimental.copies.write-to=compatibility
   $ hg changesetcopies
   files: j
+  filesadded: 
+  filesremoved: 
+  
   p1copies: 
   p2copies: 
 
@@ -131,6 +146,7 @@ 
   $ hg ci -m 'copy a to k' --config experimental.copies.write-to=filelog-only
   $ hg changesetcopies
   files: k
+  
   $ hg debugdata k 0
   \x01 (esc)
   copy: a
@@ -157,9 +173,9 @@ 
   $ hg mv a b
   $ hg ci -qm 'rename a to b'
   $ hg rebase -d 1 --config rebase.experimental.inmemory=yes
-  rebasing 2:55d0b405c1b2 "rename a to b" (tip)
+  rebasing 2:acfc33f3aa6d "rename a to b" (tip)
   merging a and b to b
-  saved backup bundle to $TESTTMP/rebase-rename/.hg/strip-backup/55d0b405c1b2-78df867e-rebase.hg
+  saved backup bundle to $TESTTMP/rebase-rename/.hg/strip-backup/acfc33f3aa6d-81d0180d-rebase.hg
   $ hg st --change . --copies
   A b
     a
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -2589,18 +2589,24 @@ 
 
         writecopiesto = self.ui.config('experimental', 'copies.write-to')
         writefilecopymeta = writecopiesto != 'changeset-only'
+        writechangesetcopy = (writecopiesto in
+                              ('changeset-only', 'compatibility'))
         p1copies, p2copies = None, None
-        if writecopiesto in ('changeset-only', 'compatibility'):
+        if writechangesetcopy:
             p1copies = ctx.p1copies()
             p2copies = ctx.p2copies()
+        filesadded, filesremoved = None, None
         with self.lock(), self.transaction("commit") as tr:
             trp = weakref.proxy(tr)
 
             if ctx.manifestnode():
                 # reuse an existing manifest revision
                 self.ui.debug('reusing known manifest\n')
                 mn = ctx.manifestnode()
                 files = ctx.files()
+                if writechangesetcopy:
+                    filesadded = ctx.filesadded()
+                    filesremoved = ctx.filesremoved()
             elif ctx.files():
                 m1ctx = p1.manifestctx()
                 m2ctx = p2.manifestctx()
@@ -2667,6 +2673,11 @@ 
                     mn = mctx.write(trp, linkrev,
                                     p1.manifestnode(), p2.manifestnode(),
                                     added, drop, match=self.narrowmatch())
+
+                    if writechangesetcopy:
+                        filesadded = [f for f in changed
+                                      if not (f in m1 or f in m2)]
+                        filesremoved = removed
                 else:
                     self.ui.debug('reusing manifest from p1 (listed files '
                                   'actually unchanged)\n')
@@ -2683,14 +2694,16 @@ 
                 # filelogs.
                 p1copies = p1copies or None
                 p2copies = p2copies or None
+                filesadded = filesadded or None
+                filesremoved = filesremoved or None
 
             # update changelog
             self.ui.note(_("committing changelog\n"))
             self.changelog.delayupdate(tr)
             n = self.changelog.add(mn, files, ctx.description(),
                                    trp, p1.node(), p2.node(),
                                    user, ctx.date(), ctx.extra().copy(),
-                                   p1copies, p2copies)
+                                   p1copies, p2copies, filesadded, filesremoved)
             xp1, xp2 = p1.hex(), p2 and p2.hex() or ''
             self.hook('pretxncommit', throw=True, node=hex(n), parent1=xp1,
                       parent2=xp2)
diff --git a/mercurial/changelog.py b/mercurial/changelog.py
--- a/mercurial/changelog.py
+++ b/mercurial/changelog.py
@@ -99,6 +99,14 @@ 
         # used different syntax for the value.
         return None
 
+def encodefileindices(files, subset):
+    subset = set(subset)
+    indices = []
+    for i, f in enumerate(files):
+        if f in subset:
+            indices.append(b'%d' % i)
+    return '\0'.join(indices)
+
 def stripdesc(desc):
     """strip trailing whitespace and leading and trailing empty lines"""
     return '\n'.join([l.rstrip() for l in desc.splitlines()]).strip('\n')
@@ -564,7 +572,8 @@ 
         return l[3:]
 
     def add(self, manifest, files, desc, transaction, p1, p2,
-                  user, date=None, extra=None, p1copies=None, p2copies=None):
+                  user, date=None, extra=None, p1copies=None, p2copies=None,
+                  filesadded=None, filesremoved=None):
         # Convert to UTF-8 encoded bytestrings as the very first
         # thing: calling any method on a localstr object will turn it
         # into a str object and the cached UTF-8 string is thus lost.
@@ -593,17 +602,23 @@ 
             elif branch in (".", "null", "tip"):
                 raise error.StorageError(_('the name \'%s\' is reserved')
                                          % branch)
-        if (p1copies is not None or p2copies is not None) and extra is None:
+        extrasentries = p1copies, p2copies, filesadded, filesremoved
+        if extra is None and any(x is not None for x in extrasentries):
             extra = {}
         if p1copies is not None:
             extra['p1copies'] = encodecopies(p1copies)
         if p2copies is not None:
             extra['p2copies'] = encodecopies(p2copies)
+        sortedfiles = sorted(files)
+        if filesadded is not None:
+            extra['filesadded'] = encodefileindices(sortedfiles, filesadded)
+        if filesremoved is not None:
+            extra['filesremoved'] = encodefileindices(sortedfiles, filesremoved)
 
         if extra:
             extra = encodeextra(extra)
             parseddate = "%s %s" % (parseddate, extra)
-        l = [hex(manifest), user, parseddate] + sorted(files) + ["", desc]
+        l = [hex(manifest), user, parseddate] + sortedfiles + ["", desc]
         text = "\n".join(l)
         return self.addrevision(text, transaction, len(self), p1, p2)