Patchwork [02,of,15] speedy: stub history query server

login
register
mail settings
Submitter Tomasz Kleczek
Date Dec. 11, 2012, 6:38 p.m.
Message ID <5d51bf818878db395e2e.1355251097@dev408.prn1.facebook.com>
Download mbox | patch
Permalink /patch/51/
State Superseded
Headers show

Comments

Tomasz Kleczek - Dec. 11, 2012, 6:38 p.m.
# HG changeset patch
# User Tomasz Kleczek <tkleczek at fb.com>
# Date 1355245443 28800
# Branch stable
# Node ID 5d51bf818878db395e2e1f840769a83254c3dcfd
# Parent  13c6bcb8dd900dc7dbf5e3da9ef68d56fed250b3
speedy: stub history query server

Introduce a server component to which clients delegate their
history requests.

For now, server instance is initialized by the client using
`makeserver` helper function and used directly from client
(all work is in the client python instance). This is going
to change in subsequent patches.

Also server still handles only author query and it simply
calls revset.author function, so no performance gain for now.

Subsequent patches are going to incrementally add functionality
to both the client and the server.
Idan Kamara - Dec. 12, 2012, 9:57 p.m.
On Tue, Dec 11, 2012 at 8:38 PM, Tomasz Kleczek <tkleczek at fb.com> wrote:
> # HG changeset patch
> # User Tomasz Kleczek <tkleczek at fb.com>
> # Date 1355245443 28800
> # Branch stable
> # Node ID 5d51bf818878db395e2e1f840769a83254c3dcfd
> # Parent  13c6bcb8dd900dc7dbf5e3da9ef68d56fed250b3
> speedy: stub history query server
>
> Introduce a server component to which clients delegate their
> history requests.

Sounds like you could use the command server for this by
adding new commands to it and making it work with TCP sockets.

Yuya already monkey patched it to work over Unix sockets so it's
worth peeking at his stuff, and in general his work on chg might
help you in some way.

https://bitbucket.org/yuja/chg
Tomasz Kleczek - Dec. 12, 2012, 10:09 p.m.
Thanks. One problem that i see right away is that command server protocol
doesn't allow you to send newlines in the data section.
Communication between speedy client and server requires sending potentially
big amount of binary data over the network and I would
like to avoid overhead such as base64 encoding..


On Wed, Dec 12, 2012 at 1:57 PM, Idan Kamara <idankk86 at gmail.com> wrote:

> On Tue, Dec 11, 2012 at 8:38 PM, Tomasz Kleczek <tkleczek at fb.com> wrote:
> > # HG changeset patch
> > # User Tomasz Kleczek <tkleczek at fb.com>
> > # Date 1355245443 28800
> > # Branch stable
> > # Node ID 5d51bf818878db395e2e1f840769a83254c3dcfd
> > # Parent  13c6bcb8dd900dc7dbf5e3da9ef68d56fed250b3
> > speedy: stub history query server
> >
> > Introduce a server component to which clients delegate their
> > history requests.
>
> Sounds like you could use the command server for this by
> adding new commands to it and making it work with TCP sockets.
>
> Yuya already monkey patched it to work over Unix sockets so it's
> worth peeking at his stuff, and in general his work on chg might
> help you in some way.
>
> https://bitbucket.org/yuja/chg
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel at selenic.com
> http://selenic.com/mailman/listinfo/mercurial-devel
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://selenic.com/pipermail/mercurial-devel/attachments/20121212/9f67e2a3/attachment.html>
Noel Grandin - Dec. 13, 2012, 6:56 a.m.
On 2012-12-13 00:09, Tomasz K?eczek wrote:
> Thanks. One problem that i see right away is that command server 
> protocol doesn't allow you to send newlines in the data section.
> Communication between speedy client and server requires sending 
> potentially big amount of binary data over the network and I would
> like to avoid overhead such as base64 encoding..
>
Maybe the protocol should be updated to have a "binary chunk" capability?
Sounds like something that would be useful for other server-side 
extensions anyway.

Disclaimer: http://www.peralex.com/disclaimer.html
Idan Kamara - Dec. 13, 2012, 9:07 a.m.
(you're top posting)

On Thu, Dec 13, 2012 at 12:09 AM, Tomasz K?eczek
<tomasz.kleczek at gmail.com> wrote:
> Thanks. One problem that i see right away is that command server protocol
> doesn't allow you to send newlines in the data section.
> Communication between speedy client and server requires sending potentially
> big amount of binary data over the network and I would
> like to avoid overhead such as base64 encoding..

The data section has no restrictions on what it contains.

You could be confused with the hello message that has a specific
format (which uses new lines to separate fields).

>
>
> On Wed, Dec 12, 2012 at 1:57 PM, Idan Kamara <idankk86 at gmail.com> wrote:
>>
>> On Tue, Dec 11, 2012 at 8:38 PM, Tomasz Kleczek <tkleczek at fb.com> wrote:
>> > # HG changeset patch
>> > # User Tomasz Kleczek <tkleczek at fb.com>
>> > # Date 1355245443 28800
>> > # Branch stable
>> > # Node ID 5d51bf818878db395e2e1f840769a83254c3dcfd
>> > # Parent  13c6bcb8dd900dc7dbf5e3da9ef68d56fed250b3
>> > speedy: stub history query server
>> >
>> > Introduce a server component to which clients delegate their
>> > history requests.
>>
>> Sounds like you could use the command server for this by
>> adding new commands to it and making it work with TCP sockets.
>>
>> Yuya already monkey patched it to work over Unix sockets so it's
>> worth peeking at his stuff, and in general his work on chg might
>> help you in some way.
>>
>> https://bitbucket.org/yuja/chg
>> _______________________________________________
>> Mercurial-devel mailing list
>> Mercurial-devel at selenic.com
>> http://selenic.com/mailman/listinfo/mercurial-devel
>
>
Kevin Bullock - Dec. 13, 2012, 3:53 p.m.
On 13 Dec 2012, at 3:07 AM, Idan Kamara wrote:

> (you're top posting)
> 
> On Thu, Dec 13, 2012 at 12:09 AM, Tomasz K?eczek
> <tomasz.kleczek at gmail.com> wrote:
>> Thanks. One problem that i see right away is that command server protocol
>> doesn't allow you to send newlines in the data section.
>> Communication between speedy client and server requires sending potentially
>> big amount of binary data over the network and I would
>> like to avoid overhead such as base64 encoding..
> 
> The data section has no restrictions on what it contains.
> 
> You could be confused with the hello message that has a specific
> format (which uses new lines to separate fields).

+1 for reusing a protocol we've already got.

pacem in terris / ??? / ?????? / ????????? / ??
Kevin R. Bullock

Patch

diff --git a/hgext/speedy/client.py b/hgext/speedy/client.py
--- a/hgext/speedy/client.py
+++ b/hgext/speedy/client.py
@@ -5,19 +5,35 @@ 
 
 from mercurial import extensions, commands
 from mercurial import revset
+from mercurial import localrepo
+from mercurial.i18n import _
+import server
 
-def patchedauthor(repo, subset, x):
+def patchedauthor(metapeer, repo, subset, x):
     """Return the revisions commited by user whose name match x
 
+    metapeer: proxy object used to query the metadata server
+
     Used to monkey patch revset.author function.
     """
-    # In the subsequent patches here we are going to forward the query
-    # to the server
-    return revset.author(repo, subset, x)
+    # We want to catch errors early and on client, if possible
+    pat = revset.getstring(x, _("author requires a string"))
+    revs = set(metapeer.author(pat))
+    return [r for r in subset if r in revs]
 
 def _speedysetup(ui, repo):
     """Initialize speedy client."""
-    revset.symbols['author'] = patchedauthor
+    # For now, local repo and server repo are the same. This is going
+    # to change soon
+    serverrepo = localrepo.localrepository(ui, path=repo.root)
+    mserver = server.makeserver(serverrepo)
+
+    def wrapwithpeer(fun, peer):
+        def wrapper(*args, **kwargs):
+            return fun(peer, *args, **kwargs)
+        return wrapper
+
+    revset.symbols['author'] = wrapwithpeer(patchedauthor, mserver)
 
 def uisetup(ui):
     # Perform patching and most of the initialization inside log wrapper,
diff --git a/hgext/speedy/server.py b/hgext/speedy/server.py
new file mode 100644
--- /dev/null
+++ b/hgext/speedy/server.py
@@ -0,0 +1,32 @@ 
+# Copyright 2012 Facebook
+#
+# This software may be used and distributed according to the terms of the
+# GNU General Public License version 2 or any later version.
+
+"""Server component
+
+metaserver: contains all the logic behind query acceleration
+"""
+
+from mercurial import revset
+
+class metaserver(object):
+    """Contains all the logic behind the query acceleration."""
+
+    def __init__(self, repo):
+        self.repo = repo
+
+    def author(self, pat):
+        """Return a list of changes commited by a user that matches pattern.
+
+        User matches pattern if their name has a `pat` substring (case
+        insensitive).
+
+        Returns a list of revs.
+        """
+        # This is going to be accelerated in the subsequent patches
+        return revset.author(self.repo, list(self.repo), ('symbol', pat))
+
+def makeserver(repo):
+    """Return an initialized metaserver instance."""
+    return metaserver(repo)
diff --git a/tests/test-speedy.t b/tests/test-speedy.t
--- a/tests/test-speedy.t
+++ b/tests/test-speedy.t
@@ -12,6 +12,11 @@ 
   $ hg init localrepo
   $ cd localrepo
 
+  $ cat > .hg/hgrc <<EOF_END
+  > [speedy]
+  > client = True
+  > EOF_END
+
   $ mkdir d1
   $ echo chg0 > d1/chg0
   $ hg commit -Am chg0 -u testuser1