Patchwork [5,of,6] upgrade: add a 'redeltafullall' mode

login
register
mail settings
Submitter Boris Feld
Date Dec. 8, 2017, 11:19 a.m.
Message ID <f020935650c9a3ba2c76.1512731959@FB>
Download mbox | patch
Permalink /patch/26100/
State Accepted
Headers show

Comments

Boris Feld - Dec. 8, 2017, 11:19 a.m.
# HG changeset patch
# User Boris Feld <boris.feld@octobus.net>
# Date 1512674823 -3600
#      Thu Dec 07 20:27:03 2017 +0100
# Node ID f020935650c9a3ba2c762524bde86243fbd39fe3
# Parent  5ac3ff23878f3a58a88540e776f9aa845ebbdd62
# EXP-Topic upgrade.config
# Available At https://bitbucket.org/octobus/mercurial-devel/
#              hg pull https://bitbucket.org/octobus/mercurial-devel/ -r f020935650c9
upgrade: add a 'redeltafullall' mode

We add a new mode for delta recomputation, when selected, each full text will
go through the full "addrevision" mechanism again. This is slower than
"redeltaall" but this gives the opportunity for extensions to trigger special
logic. For example, the lfs extensions can decide to promote some revision to
lfs storage during the upgrade.

Patch

diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -2264,7 +2264,9 @@  class revlog(object):
     DELTAREUSESAMEREVS = 'samerevs'
     DELTAREUSENEVER = 'never'
 
-    DELTAREUSEALL = {'always', 'samerevs', 'never'}
+    DELTAREUSEFULLADD = 'fulladd'
+
+    DELTAREUSEALL = {'always', 'samerevs', 'never', 'fulladd'}
 
     def clone(self, tr, destrevlog, addrevisioncb=None,
               deltareuse=DELTAREUSESAMEREVS, aggressivemergedeltas=None):
@@ -2355,18 +2357,24 @@  class revlog(object):
                 if not cachedelta:
                     rawtext = self.revision(rev, raw=True)
 
-                ifh = destrevlog.opener(destrevlog.indexfile, 'a+',
-                                        checkambig=False)
-                dfh = None
-                if not destrevlog._inline:
-                    dfh = destrevlog.opener(destrevlog.datafile, 'a+')
-                try:
-                    destrevlog._addrevision(node, rawtext, tr, linkrev, p1, p2,
-                                            flags, cachedelta, ifh, dfh)
-                finally:
-                    if dfh:
-                        dfh.close()
-                    ifh.close()
+
+                if deltareuse == self.DELTAREUSEFULLADD:
+                    destrevlog.addrevision(rawtext, tr, linkrev, p1, p2,
+                                           cachedelta=cachedelta,
+                                           node=node, flags=flags)
+                else:
+                    ifh = destrevlog.opener(destrevlog.indexfile, 'a+',
+                                            checkambig=False)
+                    dfh = None
+                    if not destrevlog._inline:
+                        dfh = destrevlog.opener(destrevlog.datafile, 'a+')
+                    try:
+                        destrevlog._addrevision(node, rawtext, tr, linkrev, p1,
+                                                p2, flags, cachedelta, ifh, dfh)
+                    finally:
+                        if dfh:
+                            dfh.close()
+                        ifh.close()
 
                 if addrevisioncb:
                     addrevisioncb(self, rev, node)
diff --git a/mercurial/upgrade.py b/mercurial/upgrade.py
--- a/mercurial/upgrade.py
+++ b/mercurial/upgrade.py
@@ -369,6 +369,19 @@  def findoptimizations(repo):
                          'recomputed; this will likely drastically slow down '
                          'execution time')))
 
+    optimizations.append(improvement(
+        name='redeltafulladd',
+        type=optimisation,
+        description=_('every revision will be re-added as if it was new '
+                      'content. It will go through the full storage '
+                      'mechanism giving extensions a chance to process it '
+                      '(eg. lfs). This is similar to "redeltaall" but even '
+                      'slower since more logic is involved.'),
+        upgrademessage=_('each revision will be added as new content to the '
+                         'internal storage; this will likely drastically slow '
+                         'down execution time, but some extensions might need '
+                         'it')))
+
     return optimizations
 
 def determineactions(repo, deficiencies, sourcereqs, destreqs):
@@ -618,6 +631,8 @@  def _upgraderepo(ui, srcrepo, dstrepo, r
         deltareuse = revlog.revlog.DELTAREUSESAMEREVS
     elif 'redeltamultibase' in actions:
         deltareuse = revlog.revlog.DELTAREUSESAMEREVS
+    if 'redeltafulladd' in actions:
+        deltareuse = revlog.revlog.DELTAREUSEFULLADD
     else:
         deltareuse = revlog.revlog.DELTAREUSEALWAYS
 
diff --git a/tests/test-upgrade-repo.t b/tests/test-upgrade-repo.t
--- a/tests/test-upgrade-repo.t
+++ b/tests/test-upgrade-repo.t
@@ -100,6 +100,9 @@  An upgrade of a repository created with 
   redeltaall
      deltas within internal storage will always be recalculated without reusing prior deltas; this will likely make execution run several times slower; this optimization is typically not needed
   
+  redeltafulladd
+     every revision will be re-added as if it was new content. It will go through the full storage mechanism giving extensions a chance to process it (eg. lfs). This is similar to "redeltaall" but even slower since more logic is involved.
+  
 
 --optimize can be used to add optimizations
 
@@ -121,6 +124,9 @@  An upgrade of a repository created with 
   redeltaall
      deltas within internal storage will always be recalculated without reusing prior deltas; this will likely make execution run several times slower; this optimization is typically not needed
   
+  redeltafulladd
+     every revision will be re-added as if it was new content. It will go through the full storage mechanism giving extensions a chance to process it (eg. lfs). This is similar to "redeltaall" but even slower since more logic is involved.
+  
 
 Various sub-optimal detections work
 
@@ -196,6 +202,9 @@  Various sub-optimal detections work
   redeltaall
      deltas within internal storage will always be recalculated without reusing prior deltas; this will likely make execution run several times slower; this optimization is typically not needed
   
+  redeltafulladd
+     every revision will be re-added as if it was new content. It will go through the full storage mechanism giving extensions a chance to process it (eg. lfs). This is similar to "redeltaall" but even slower since more logic is involved.
+  
 
   $ hg --config format.dotencode=false debugupgraderepo
   repository lacks features recommended by current config options:
@@ -235,6 +244,9 @@  Various sub-optimal detections work
   redeltaall
      deltas within internal storage will always be recalculated without reusing prior deltas; this will likely make execution run several times slower; this optimization is typically not needed
   
+  redeltafulladd
+     every revision will be re-added as if it was new content. It will go through the full storage mechanism giving extensions a chance to process it (eg. lfs). This is similar to "redeltaall" but even slower since more logic is involved.
+  
 
   $ cd ..
 
@@ -406,6 +418,40 @@  store files with special filenames aren'
   removing temporary repository $TESTTMP/store-filenames/.hg/upgrade.* (glob)
   copy of old repository backed up at $TESTTMP/store-filenames/.hg/upgradebackup.* (glob)
   the old repository will not be deleted; remove it to free up disk space once the upgraded repository is verified
+  $ hg debugupgraderepo --run --optimize redeltafulladd
+  upgrade will perform the following actions:
+  
+  requirements
+     preserved: dotencode, fncache, generaldelta, revlogv1, store
+  
+  redeltafulladd
+     each revision will be added as new content to the internal storage; this will likely drastically slow down execution time, but some extensions might need it
+  
+  beginning upgrade...
+  repository locked and read-only
+  creating temporary repository to stage migrated data: $TESTTMP/store-filenames/.hg/upgrade.* (glob)
+  (it is safe to interrupt this process any time before data migration completes)
+  migrating 3 total revisions (1 in filelogs, 1 in manifests, 1 in changelog)
+  migrating 109 bytes in store; 107 bytes tracked data
+  migrating 1 filelogs containing 1 revisions (0 bytes in store; 0 bytes tracked data)
+  finished migrating 1 filelog revisions across 1 filelogs; change in size: 0 bytes
+  migrating 1 manifests containing 1 revisions (46 bytes in store; 45 bytes tracked data)
+  finished migrating 1 manifest revisions across 1 manifests; change in size: 0 bytes
+  migrating changelog containing 1 revisions (63 bytes in store; 62 bytes tracked data)
+  finished migrating 1 changelog revisions; change in size: 0 bytes
+  finished migrating 3 total revisions; total change in store size: 0 bytes
+  copying .XX_special_filename
+  copying phaseroots
+  data fully migrated to temporary repository
+  marking source repository as being upgraded; clients will be unable to read from repository
+  starting in-place swap of repository data
+  replaced files will be backed up at $TESTTMP/store-filenames/.hg/upgradebackup.* (glob)
+  replacing store...
+  store replacement complete; repository was inconsistent for 0.0s
+  finalizing requirements file and making repository readable again
+  removing temporary repository $TESTTMP/store-filenames/.hg/upgrade.* (glob)
+  copy of old repository backed up at $TESTTMP/store-filenames/.hg/upgradebackup.* (glob)
+  the old repository will not be deleted; remove it to free up disk space once the upgraded repository is verified
 
   $ cd ..