Patchwork [3,of,3] revlog-sparse-read: add a lower-threshold for read block size

login
register
mail settings
Submitter Paul Morelle
Date Oct. 16, 2017, 4:35 p.m.
Message ID <cac7666538643b087500.1508171750@taranis.localdomain>
Download mbox | patch
Permalink /patch/25004/
State Accepted
Headers show

Comments

Paul Morelle - Oct. 16, 2017, 4:35 p.m.
# HG changeset patch
# User Paul Morelle <paul.morelle@octobus.net>
# Date 1507993541 -7200
#      Sat Oct 14 17:05:41 2017 +0200
# Node ID cac7666538643b087500c09c757a1b5f28421fbb
# Parent  fd6ea10467600ccdfc9f3491ad95da5cdb5b840d
# EXP-Topic optimized-read
# Available At https://bitbucket.org/octobus/mercurial-devel/
#              hg pull https://bitbucket.org/octobus/mercurial-devel/ -r cac766653864
revlog-sparse-read: add a lower-threshold for read block size

The option experimental.sparse-read.min-block-size specifies the minimal size
of a deltachain span, under which it won't be split by _slicechunk.
Augie Fackler - Oct. 17, 2017, 3:22 p.m.
On Mon, Oct 16, 2017 at 06:35:50PM +0200, Paul Morelle wrote:
> # HG changeset patch
> # User Paul Morelle <paul.morelle@octobus.net>
> # Date 1507993541 -7200
> #      Sat Oct 14 17:05:41 2017 +0200
> # Node ID cac7666538643b087500c09c757a1b5f28421fbb
> # Parent  fd6ea10467600ccdfc9f3491ad95da5cdb5b840d
> # EXP-Topic optimized-read
> # Available At https://bitbucket.org/octobus/mercurial-devel/
> #              hg pull https://bitbucket.org/octobus/mercurial-devel/ -r cac766653864
> revlog-sparse-read: add a lower-threshold for read block size

queued, thanks

I'm a little nervous about bugs in this area, but it seems simple
enough and is behind a config knob so it's easy for us to drop the
whole feature in 4.4.1 if it causes problems.

Patch

diff -r fd6ea1046760 -r cac766653864 mercurial/configitems.py
--- a/mercurial/configitems.py	Tue Oct 10 17:50:27 2017 +0200
+++ b/mercurial/configitems.py	Sat Oct 14 17:05:41 2017 +0200
@@ -415,6 +415,9 @@ 
 coreconfigitem('experimental', 'sparse-read.density-threshold',
     default=0.25,
 )
+coreconfigitem('experimental', 'sparse-read.min-block-size',
+    default='256K',
+)
 coreconfigitem('experimental', 'treemanifest',
     default=False,
 )
diff -r fd6ea1046760 -r cac766653864 mercurial/localrepo.py
--- a/mercurial/localrepo.py	Tue Oct 10 17:50:27 2017 +0200
+++ b/mercurial/localrepo.py	Sat Oct 14 17:05:41 2017 +0200
@@ -611,8 +611,11 @@ 
         withsparseread = self.ui.configbool('experimental', 'sparse-read')
         srdensitythres = float(self.ui.config('experimental',
                                               'sparse-read.density-threshold'))
+        srminblocksize = self.ui.configbytes('experimental',
+                                             'sparse-read.min-block-size')
         self.svfs.options['with-sparse-read'] = withsparseread
         self.svfs.options['sparse-read-density-threshold'] = srdensitythres
+        self.svfs.options['sparse-read-min-block-size'] = srminblocksize
 
         for r in self.requirements:
             if r.startswith('exp-compression-'):
diff -r fd6ea1046760 -r cac766653864 mercurial/revlog.py
--- a/mercurial/revlog.py	Tue Oct 10 17:50:27 2017 +0200
+++ b/mercurial/revlog.py	Sat Oct 14 17:05:41 2017 +0200
@@ -180,7 +180,7 @@ 
         endbyte = start(revs[-1]) + length(revs[-1])
         deltachainspan = endbyte - startbyte
 
-        if len(revs) <= 1:
+        if deltachainspan <= revlog._srminblocksize or len(revs) <= 1:
             yield revs
             continue
 
@@ -359,6 +359,7 @@ 
         self._maxdeltachainspan = -1
         self._withsparseread = False
         self._srdensitythreshold = 0.25
+        self._srminblocksize = 262144
 
         mmapindexthreshold = None
         v = REVLOG_DEFAULT_VERSION
@@ -388,6 +389,8 @@ 
             self._withsparseread = bool(opts.get('with-sparse-read', False))
             if 'sparse-read-density-threshold' in opts:
                 self._srdensitythreshold = opts['sparse-read-density-threshold']
+            if 'sparse-read-min-block-size' in opts:
+                self._srminblocksize = opts['sparse-read-min-block-size']
 
         if self._chunkcachesize <= 0:
             raise RevlogError(_('revlog chunk cache size %r is not greater '