Patchwork [2,of,2,V3,mainline] format: introduce 'format.usegeneraldelta`

login
register
mail settings
Submitter Pierre-Yves David
Date Nov. 11, 2015, 1:55 a.m.
Message ID <c94b9591b95fe6580f5d.1447206905@marginatus.alto.octopoid.net>
Download mbox | patch
Permalink /patch/11363/
State Accepted
Headers show

Comments

Pierre-Yves David - Nov. 11, 2015, 1:55 a.m.
# HG changeset patch
# User Pierre-Yves David <pierre-yves.david@fb.com>
# Date 1446479952 0
#      Mon Nov 02 15:59:12 2015 +0000
# Node ID c94b9591b95fe6580f5d8455da2c90806d8ccf28
# Parent  a21c430969a1159c590394b4f77301e2675f191a
# EXP-Topic generaldelta
# Available At http://hg.netv6.net/marmoute-wip/mercurial/
#              hg pull http://hg.netv6.net/marmoute-wip/mercurial/ -r c94b9591b95f
format: introduce 'format.usegeneraldelta`

This option will make repository created as general delta by default but will
not make Mercurial aggressively recompute delta for all incoming bundle.
Instead, the delta contained in the bundle will be used. This will allow use to
start having general delta repository created everywhere without triggering
massive re-computation cost for all new clients cloning from old server.
Matt Mackall - Nov. 11, 2015, 10:59 p.m.
On Tue, 2015-11-10 at 17:55 -0800, Pierre-Yves David wrote:
> # HG changeset patch
> # User Pierre-Yves David <pierre-yves.david@fb.com>
> # Date 1446479952 0
> #      Mon Nov 02 15:59:12 2015 +0000
> # Node ID c94b9591b95fe6580f5d8455da2c90806d8ccf28
> # Parent  a21c430969a1159c590394b4f77301e2675f191a
> # EXP-Topic generaldelta
> # Available At http://hg.netv6.net/marmoute-wip/mercurial/
> #              hg pull http://hg.netv6.net/marmoute-wip/mercurial/ -r
> c94b9591b95f
> format: introduce 'format.usegeneraldelta`

These are queued for default, thanks.

-- 
Mathematics is the supreme nostalgia of our time.
Gregory Szorc - Nov. 11, 2015, 11:38 p.m.
On Tue, Nov 10, 2015 at 5:55 PM, Pierre-Yves David <
pierre-yves.david@ens-lyon.org> wrote:

> # HG changeset patch
> # User Pierre-Yves David <pierre-yves.david@fb.com>
> # Date 1446479952 0
> #      Mon Nov 02 15:59:12 2015 +0000
> # Node ID c94b9591b95fe6580f5d8455da2c90806d8ccf28
> # Parent  a21c430969a1159c590394b4f77301e2675f191a
> # EXP-Topic generaldelta
> # Available At http://hg.netv6.net/marmoute-wip/mercurial/
> #              hg pull http://hg.netv6.net/marmoute-wip/mercurial/ -r
> c94b9591b95f
> format: introduce 'format.usegeneraldelta`
>
> This option will make repository created as general delta by default but
> will
> not make Mercurial aggressively recompute delta for all incoming bundle.
> Instead, the delta contained in the bundle will be used. This will allow
> use to
> start having general delta repository created everywhere without triggering
> massive re-computation cost for all new clients cloning from old server.
>
> diff --git a/mercurial/help/config.txt b/mercurial/help/config.txt
> --- a/mercurial/help/config.txt
> +++ b/mercurial/help/config.txt
> @@ -664,10 +664,18 @@ Example for ``~/.hgrc``::
>
>
>  ``format``
>  ----------
>
> +``usegeneraldelta``
> +    Enable or disable the "generaldelta" repository format which improve
> +    repository compression by allowing "revlog" to store delta against
> arbitrary
> +    revision instead of the previous stored one. This provide significant
> +    improvement for repository with branches. Disabling this option
> ensure that
> +    on-disk format of newly created repository will be compatible with
> Mercurial
> +    before version 1.9.
> +
>  ``usestore``
>      Enable or disable the "store" repository format which improves
>      compatibility with systems that fold case or otherwise mangle
>      filenames. Enabled by default. Disabling this option will allow
>      you to store longer filenames in some situations at the expense of
> diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
> --- a/mercurial/localrepo.py
> +++ b/mercurial/localrepo.py
> @@ -356,10 +356,11 @@ class localrepository(object):
>              self.svfs.options['manifestcachesize'] = manifestcachesize
>          # experimental config: format.aggressivemergedeltas
>          aggressivemergedeltas = self.ui.configbool('format',
>              'aggressivemergedeltas', False)
>          self.svfs.options['aggressivemergedeltas'] = aggressivemergedeltas
> +        self.svfs.options['lazydeltabase'] = not
> scmutil.gddeltaconfig(self.ui)
>
>      def _writerequirements(self):
>          scmutil.writerequires(self.vfs, self.requirements)
>
>      def _checknested(self, path):
> diff --git a/mercurial/revlog.py b/mercurial/revlog.py
> --- a/mercurial/revlog.py
> +++ b/mercurial/revlog.py
> @@ -228,10 +228,11 @@ class revlog(object):
>                  self._chunkcachesize = opts['chunkcachesize']
>              if 'maxchainlen' in opts:
>                  self._maxchainlen = opts['maxchainlen']
>              if 'aggressivemergedeltas' in opts:
>                  self._aggressivemergedeltas =
> opts['aggressivemergedeltas']
> +            self._lazydeltabase = bool(opts.get('lazydeltabase', False))
>
>          if self._chunkcachesize <= 0:
>              raise RevlogError(_('revlog chunk cache size %r is not
> greater '
>                                  'than 0') % self._chunkcachesize)
>          elif self._chunkcachesize & (self._chunkcachesize - 1):
> @@ -1368,11 +1369,15 @@ class revlog(object):
>          else:
>              textlen = len(text)
>
>          # should we try to build a delta?
>          if prev != nullrev:
> -            if self._generaldelta:
> +            if cachedelta and self._generaldelta and self._lazydeltabase:
> +                # Assume what we received from the server is a good choice
> +                # build delta will reuse the cache
> +                d = builddelta(cachedelta[0])
> +            elif self._generaldelta:
>                  if p2r != nullrev and self._aggressivemergedeltas:
>                      d = builddelta(p1r)
>                      d2 = builddelta(p2r)
>                      p1good = self._isgooddelta(d, textlen)
>                      p2good = self._isgooddelta(d2, textlen)
> diff --git a/mercurial/scmutil.py b/mercurial/scmutil.py
> --- a/mercurial/scmutil.py
> +++ b/mercurial/scmutil.py
> @@ -1171,10 +1171,15 @@ def wlocksub(repo, cmd, *args, **kwargs)
>      return _locksub(repo, repo.currentwlock(), 'HG_WLOCK_LOCKER', cmd,
> *args,
>                      **kwargs)
>
>  def gdinitconfig(ui):
>      """helper function to know if a repo should be created as general
> delta
> +    """
> +    # experimental config: format.generaldelta
> +    return (ui.configbool('format', 'generaldelta', False)
> +            or ui.configbool('format', 'usegeneraldelta', False))
>
> -    This currently depends on a single config option but this will get
> more
> -    complicated soon."""
> +def gddeltaconfig(ui):
> +    """helper function to know if incoming delta should be optimised
> +    """
>      # experimental config: format.generaldelta
>      return ui.configbool('format', 'generaldelta', False)
> diff --git a/tests/test-generaldelta.t b/tests/test-generaldelta.t
> --- a/tests/test-generaldelta.t
> +++ b/tests/test-generaldelta.t
> @@ -69,14 +69,59 @@ commit.
>       rev    offset  length   base linkrev nodeid       p1           p2
>         0         0       3      0       1 1406e7411862 000000000000
> 000000000000
>
>    $ cd ..
>
> +Test "usegeneraldelta" config
> +(repo are general delta, but incoming bundle are not re-deltified)
> +
> +delta coming from the server base delta server are not recompressed.
> +(also include the aggressive version for comparison)
> +
> +  $ hg clone repo --pull --config format.usegeneraldelta=1 usegd
> +  requesting all changes
> +  adding changesets
> +  adding manifests
> +  adding file changes
> +  added 4 changesets with 5 changes to 2 files (+2 heads)
> +  updating to branch default
> +  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
> +  $ hg clone repo --pull --config format.generaldelta=1 full
> +  requesting all changes
> +  adding changesets
> +  adding manifests
> +  adding file changes
> +  added 4 changesets with 5 changes to 2 files (+2 heads)
> +  updating to branch default
> +  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
>

Should we throw in some checks of .hg/requires and `hg debugrevlog -m` to
ensure the expected generaldelta values are present?


> +  $ hg -R repo debugindex -m
> +     rev    offset  length   base linkrev nodeid       p1           p2
> +       0         0      77      0       0 0273e8a1b972 000000000000
> 000000000000
> +       1        77      57      0       1 e0c49f5ef780 0273e8a1b972
> 000000000000
> +       2       134      77      2       2 de950093e41b 0273e8a1b972
> 000000000000
> +       3       211      57      2       3 db74c7cde4d0 0273e8a1b972
> 000000000000
> +  $ hg -R usegd debugindex -m
> +     rev    offset  length  delta linkrev nodeid       p1           p2
> +       0         0      77     -1       0 0273e8a1b972 000000000000
> 000000000000
> +       1        77      57      0       1 e0c49f5ef780 0273e8a1b972
> 000000000000
> +       2       134      77     -1       2 de950093e41b 0273e8a1b972
> 000000000000
> +       3       211      57      2       3 db74c7cde4d0 0273e8a1b972
> 000000000000
> +  $ hg -R full debugindex -m
> +     rev    offset  length  delta linkrev nodeid       p1           p2
> +       0         0      77     -1       0 0273e8a1b972 000000000000
> 000000000000
> +       1        77      57      0       1 e0c49f5ef780 0273e8a1b972
> 000000000000
> +       2       134      57      0       2 de950093e41b 0273e8a1b972
> 000000000000
> +       3       191      57      0       3 db74c7cde4d0 0273e8a1b972
> 000000000000
> +
>  Test format.aggressivemergedeltas
>
>    $ hg init --config format.generaldelta=1 aggressive
>    $ cd aggressive
> +  $ cat << EOF >> .hg/hgrc
> +  > [format]
> +  > generaldelta = 1
> +  > EOF
>    $ touch a b c d e
>    $ hg commit -Aqm side1
>    $ hg up -q null
>    $ touch x y
>    $ hg commit -Aqm side2
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@selenic.com
> https://selenic.com/mailman/listinfo/mercurial-devel
>

Patch

diff --git a/mercurial/help/config.txt b/mercurial/help/config.txt
--- a/mercurial/help/config.txt
+++ b/mercurial/help/config.txt
@@ -664,10 +664,18 @@  Example for ``~/.hgrc``::
 
 
 ``format``
 ----------
 
+``usegeneraldelta``
+    Enable or disable the "generaldelta" repository format which improve
+    repository compression by allowing "revlog" to store delta against arbitrary
+    revision instead of the previous stored one. This provide significant
+    improvement for repository with branches. Disabling this option ensure that
+    on-disk format of newly created repository will be compatible with Mercurial
+    before version 1.9.
+
 ``usestore``
     Enable or disable the "store" repository format which improves
     compatibility with systems that fold case or otherwise mangle
     filenames. Enabled by default. Disabling this option will allow
     you to store longer filenames in some situations at the expense of
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -356,10 +356,11 @@  class localrepository(object):
             self.svfs.options['manifestcachesize'] = manifestcachesize
         # experimental config: format.aggressivemergedeltas
         aggressivemergedeltas = self.ui.configbool('format',
             'aggressivemergedeltas', False)
         self.svfs.options['aggressivemergedeltas'] = aggressivemergedeltas
+        self.svfs.options['lazydeltabase'] = not scmutil.gddeltaconfig(self.ui)
 
     def _writerequirements(self):
         scmutil.writerequires(self.vfs, self.requirements)
 
     def _checknested(self, path):
diff --git a/mercurial/revlog.py b/mercurial/revlog.py
--- a/mercurial/revlog.py
+++ b/mercurial/revlog.py
@@ -228,10 +228,11 @@  class revlog(object):
                 self._chunkcachesize = opts['chunkcachesize']
             if 'maxchainlen' in opts:
                 self._maxchainlen = opts['maxchainlen']
             if 'aggressivemergedeltas' in opts:
                 self._aggressivemergedeltas = opts['aggressivemergedeltas']
+            self._lazydeltabase = bool(opts.get('lazydeltabase', False))
 
         if self._chunkcachesize <= 0:
             raise RevlogError(_('revlog chunk cache size %r is not greater '
                                 'than 0') % self._chunkcachesize)
         elif self._chunkcachesize & (self._chunkcachesize - 1):
@@ -1368,11 +1369,15 @@  class revlog(object):
         else:
             textlen = len(text)
 
         # should we try to build a delta?
         if prev != nullrev:
-            if self._generaldelta:
+            if cachedelta and self._generaldelta and self._lazydeltabase:
+                # Assume what we received from the server is a good choice
+                # build delta will reuse the cache
+                d = builddelta(cachedelta[0])
+            elif self._generaldelta:
                 if p2r != nullrev and self._aggressivemergedeltas:
                     d = builddelta(p1r)
                     d2 = builddelta(p2r)
                     p1good = self._isgooddelta(d, textlen)
                     p2good = self._isgooddelta(d2, textlen)
diff --git a/mercurial/scmutil.py b/mercurial/scmutil.py
--- a/mercurial/scmutil.py
+++ b/mercurial/scmutil.py
@@ -1171,10 +1171,15 @@  def wlocksub(repo, cmd, *args, **kwargs)
     return _locksub(repo, repo.currentwlock(), 'HG_WLOCK_LOCKER', cmd, *args,
                     **kwargs)
 
 def gdinitconfig(ui):
     """helper function to know if a repo should be created as general delta
+    """
+    # experimental config: format.generaldelta
+    return (ui.configbool('format', 'generaldelta', False)
+            or ui.configbool('format', 'usegeneraldelta', False))
 
-    This currently depends on a single config option but this will get more
-    complicated soon."""
+def gddeltaconfig(ui):
+    """helper function to know if incoming delta should be optimised
+    """
     # experimental config: format.generaldelta
     return ui.configbool('format', 'generaldelta', False)
diff --git a/tests/test-generaldelta.t b/tests/test-generaldelta.t
--- a/tests/test-generaldelta.t
+++ b/tests/test-generaldelta.t
@@ -69,14 +69,59 @@  commit.
      rev    offset  length   base linkrev nodeid       p1           p2
        0         0       3      0       1 1406e7411862 000000000000 000000000000
 
   $ cd ..
 
+Test "usegeneraldelta" config
+(repo are general delta, but incoming bundle are not re-deltified)
+
+delta coming from the server base delta server are not recompressed.
+(also include the aggressive version for comparison)
+
+  $ hg clone repo --pull --config format.usegeneraldelta=1 usegd
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 4 changesets with 5 changes to 2 files (+2 heads)
+  updating to branch default
+  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ hg clone repo --pull --config format.generaldelta=1 full
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 4 changesets with 5 changes to 2 files (+2 heads)
+  updating to branch default
+  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ hg -R repo debugindex -m
+     rev    offset  length   base linkrev nodeid       p1           p2
+       0         0      77      0       0 0273e8a1b972 000000000000 000000000000
+       1        77      57      0       1 e0c49f5ef780 0273e8a1b972 000000000000
+       2       134      77      2       2 de950093e41b 0273e8a1b972 000000000000
+       3       211      57      2       3 db74c7cde4d0 0273e8a1b972 000000000000
+  $ hg -R usegd debugindex -m
+     rev    offset  length  delta linkrev nodeid       p1           p2
+       0         0      77     -1       0 0273e8a1b972 000000000000 000000000000
+       1        77      57      0       1 e0c49f5ef780 0273e8a1b972 000000000000
+       2       134      77     -1       2 de950093e41b 0273e8a1b972 000000000000
+       3       211      57      2       3 db74c7cde4d0 0273e8a1b972 000000000000
+  $ hg -R full debugindex -m
+     rev    offset  length  delta linkrev nodeid       p1           p2
+       0         0      77     -1       0 0273e8a1b972 000000000000 000000000000
+       1        77      57      0       1 e0c49f5ef780 0273e8a1b972 000000000000
+       2       134      57      0       2 de950093e41b 0273e8a1b972 000000000000
+       3       191      57      0       3 db74c7cde4d0 0273e8a1b972 000000000000
+
 Test format.aggressivemergedeltas
 
   $ hg init --config format.generaldelta=1 aggressive
   $ cd aggressive
+  $ cat << EOF >> .hg/hgrc
+  > [format]
+  > generaldelta = 1
+  > EOF
   $ touch a b c d e
   $ hg commit -Aqm side1
   $ hg up -q null
   $ touch x y
   $ hg commit -Aqm side2