Patchwork [remotefilelog-ext] fileserverclient: use new iterbatch() method

login
register
mail settings
Submitter Durham Goode
Date March 22, 2016, 5:07 p.m.
Message ID <D5DA55EB-C589-42E7-ACD4-DE15938EDA8F@fb.com>
Download mbox | patch
Permalink /patch/14026/
State Accepted
Headers show

Comments

Durham Goode - March 22, 2016, 5:07 p.m.
Accepted and pushed (with Martin's nit fixed). Thanks!

From: Martin von Zweigbergk <martinvonz@google.com<mailto:martinvonz@google.com>>

Date: Sunday, March 20, 2016 at 8:46 PM
To: Augie Fackler <raf@durin42.com<mailto:raf@durin42.com>>, Durham Goode <durham@fb.com<mailto:durham@fb.com>>
Cc: "mercurial-devel@mercurial-scm.org<mailto:mercurial-devel@mercurial-scm.org>" <mercurial-devel@mercurial-scm.org<mailto:mercurial-devel@mercurial-scm.org>>
Subject: Re: [PATCH remotefilelog-ext] fileserverclient: use new iterbatch() method

Nit for Durham to fix in flight: drop the "not" in "isn't streamy not because batch() wasn't"

On Sun, Mar 20, 2016, 19:51 Augie Fackler <raf@durin42.com<mailto:raf@durin42.com>> wrote:
# HG changeset patch
# User Augie Fackler <augie@google.com<mailto:augie@google.com>>
# Date 1457036127 18000
#      Thu Mar 03 15:15:27 2016 -0500
# Node ID 669622eff4bfc2273a0bd7fd102819fabf0fc03c
# Parent  be02547f644b229f84bdeb47970fbb9198741695
fileserverclient: use new iterbatch() method

This allows the client to send a single batch request for all file contents
and then handle the responses as they stream back to the client, which should
improve both running time and the user experience as far as it goes with
progress.

_______________________________________________
Mercurial-devel mailing list
Mercurial-devel@mercurial-scm.org<mailto:Mercurial-devel@mercurial-scm.org>
https://www.mercurial-scm.org/mailman/listinfo/mercurial-devel<https://urldefense.proofpoint.com/v2/url?u=https-3A__www.mercurial-2Dscm.org_mailman_listinfo_mercurial-2Ddevel&d=CwMFaQ&c=5VD0RTtNlTh3ycd41b3MUw&r=nuarHzhP1wi1T9iURRCj1A&m=hOqR6hvZ019DDGKILl1eQDwGxMGEOnr5xiT7KO5r-Fw&s=a7fqnGtEsFvz0jmVYFIw-1sIJ3lfaLxK6u8fqmKa_NU&e=>

Patch

diff --git a/remotefilelog/fileserverclient.py b/remotefilelog/fileserverclient.py

--- a/remotefilelog/fileserverclient.py

+++ b/remotefilelog/fileserverclient.py

@@ -6,9 +6,10 @@ 

 # GNU General Public License version 2 or any later version.

 from mercurial.i18n import _
-from mercurial import util, sshpeer, hg, error, util, wireproto, node

+from mercurial import util, sshpeer, hg, error, util, wireproto, node, httppeer

 import os, socket, lz4, time, grp, io
 import errno
+import itertools


 # Statistics for debugging
 fetchcost = 0
@@ -121,6 +122,28 @@  class cacheconnection(object):


 def _getfilesbatch(
         remote, receivemissing, progresstick, missed, idmap, batchsize):
+    # Over http(s), iterbatch is a streamy method and we can start

+    # looking at results early. This means we send one (potentially

+    # large) request, but then we show nice progress as we process

+    # file results, rather than showing chunks of $batchsize in

+    # progress.

+    #

+    # Over ssh, iterbatch isn't streamy not because batch() wasn't

+    # explicitly designed as a streaming method. In the future we

+    # should probably introduce a streambatch() method upstream and

+    # use that for this.

+    if (getattr(remote, 'iterbatch', False) and remote.capable('httppostargs')

+        and isinstance(remote, httppeer.httppeer)):

+        b = remote.iterbatch()

+        for m in missed:

+            file_ = idmap[m]

+            node = m[-40:]

+            b.getfile(file_, node)

+        b.submit()

+        for m, r in itertools.izip(missed, b.results()):

+            receivemissing(io.BytesIO('%d\n%s' % (len(r), r)), m)

+            progresstick()

+        return

     while missed:
         chunk, missed = missed[:batchsize], missed[batchsize:]
         b = remote.batch()