Patchwork [5,of,5,remotefilelog-http] fileserverclient: mark getfile as batchable

login
register
mail settings
Submitter Augie Fackler
Date July 16, 2015, 3:36 p.m.
Message ID <06b2fcdc29bf96545955.1437061009@imladris.local>
Download mbox | patch
Permalink /patch/10006/
State Accepted
Headers show

Comments

Augie Fackler - July 16, 2015, 3:36 p.m.
# HG changeset patch
# User Augie Fackler <augie@google.com>
# Date 1435700041 14400
#      Tue Jun 30 17:34:01 2015 -0400
# Node ID 06b2fcdc29bf965459555d09141ffc27f6ee9f8f
# Parent  ea8c6483a58df0eb0c296abc17cc846be80ed8cb
fileserverclient: mark getfile as batchable

This lets clients send many getfile requests in a single transaction.

Note that this requires d3d32643c060 be applied to your Mercurial, or
you'll be bitten by a bug[0] in Mercurial's wireproto batching. As a
result of this change, remotefilelog now effectively requires the
upcoming Mercurial 3.5 if you want to use a specific release.

0: http://bz.selenic.com/show_bug.cgi?id=4739
Durham Goode - July 16, 2015, 5:41 p.m.
Series looks good.  Will queue once I hear back about Patch 4

On 7/16/15, 8:36 AM, "raf@durin42.com" <raf@durin42.com> wrote:

># HG changeset patch
># User Augie Fackler <augie@google.com>
># Date 1435700041 14400
>#      Tue Jun 30 17:34:01 2015 -0400
># Node ID 06b2fcdc29bf965459555d09141ffc27f6ee9f8f
># Parent  ea8c6483a58df0eb0c296abc17cc846be80ed8cb
>fileserverclient: mark getfile as batchable
>
>This lets clients send many getfile requests in a single transaction.
>
>Note that this requires d3d32643c060 be applied to your Mercurial, or
>you'll be bitten by a bug[0] in Mercurial's wireproto batching. As a
>result of this change, remotefilelog now effectively requires the
>upcoming Mercurial 3.5 if you want to use a specific release.
>
>0: http://bz.selenic.com/show_bug.cgi?id=4739
>
>diff --git a/remotefilelog/fileserverclient.py
>b/remotefilelog/fileserverclient.py
>--- a/remotefilelog/fileserverclient.py
>+++ b/remotefilelog/fileserverclient.py
>@@ -36,8 +36,14 @@ def getlocalkey(file, id):
> 
> def peersetup(ui, peer):
>     class remotefilepeer(peer.__class__):
>+        @wireproto.batchable
>         def getfile(self, file, node):
>-            return self._call('getfile', file=file, node=node)
>+            if not self.capable('getfile'):
>+                raise util.Abort(
>+                    'configured remotefile server does not support
>getfile')
>+            f = wireproto.future()
>+            yield {'file': file, 'node': node}, f
>+            yield f.value
>     peer.__class__ = remotefilepeer
> 
> class cacheconnection(object):
>@@ -92,10 +98,15 @@ class cacheconnection(object):
>         return result
> 
> def _getfilesbatch(remote, receivemissing, progresstick, missed, idmap):
>+    b = remote.batch()
>+    futures = {}
>     for m in missed:
>         file_ = idmap[m]
>         node = m[-40:]
>-        v = remote.getfile(file_, node)
>+        futures[m] = b.getfile(file_, node)
>+    b.submit()
>+    for m in missed:
>+        v = futures[m].value
>         receivemissing(io.BytesIO('%d\n%s' % (len(v), v)), m)
>         progresstick()
>

Patch

diff --git a/remotefilelog/fileserverclient.py b/remotefilelog/fileserverclient.py
--- a/remotefilelog/fileserverclient.py
+++ b/remotefilelog/fileserverclient.py
@@ -36,8 +36,14 @@  def getlocalkey(file, id):
 
 def peersetup(ui, peer):
     class remotefilepeer(peer.__class__):
+        @wireproto.batchable
         def getfile(self, file, node):
-            return self._call('getfile', file=file, node=node)
+            if not self.capable('getfile'):
+                raise util.Abort(
+                    'configured remotefile server does not support getfile')
+            f = wireproto.future()
+            yield {'file': file, 'node': node}, f
+            yield f.value
     peer.__class__ = remotefilepeer
 
 class cacheconnection(object):
@@ -92,10 +98,15 @@  class cacheconnection(object):
         return result
 
 def _getfilesbatch(remote, receivemissing, progresstick, missed, idmap):
+    b = remote.batch()
+    futures = {}
     for m in missed:
         file_ = idmap[m]
         node = m[-40:]
-        v = remote.getfile(file_, node)
+        futures[m] = b.getfile(file_, node)
+    b.submit()
+    for m in missed:
+        v = futures[m].value
         receivemissing(io.BytesIO('%d\n%s' % (len(v), v)), m)
         progresstick()