Patchwork [STABLE] hgweb: cache fctx.parents() in annotate command (issue5414)

login
register
mail settings
Submitter Gregory Szorc
Date Nov. 5, 2016, 6:07 a.m.
Message ID <b0dc7dd87fc8b5efee6e.1478326041@ubuntu-vm-main>
Download mbox | patch
Permalink /patch/17349/
State Superseded
Headers show

Comments

Gregory Szorc - Nov. 5, 2016, 6:07 a.m.
# HG changeset patch
# User Gregory Szorc <gregory.szorc@gmail.com>
# Date 1478325927 25200
#      Fri Nov 04 23:05:27 2016 -0700
# Branch stable
# Node ID b0dc7dd87fc8b5efee6e6d8c0bcc6ddfa52d6df3
# Parent  e5cc44ea12de681d971fcbebb65a7fb71fd1c3c7
hgweb: cache fctx.parents() in annotate command (issue5414)

9c37df347485 introduced a call to fctx.parents() for each line in
annotate output. This function call isn't cheap, as it requires
linkrev adjustment.

Since multiple lines in annotate output tend to belong to the same
file revision, a cache of fctx.parents() lookups for each input
should be effective in the common case. So we implement one.

The effect of this change when requesting /annotate/96ca0ecdcfa/
browser/locales/en-US/chrome/browser/downloads/downloads.dtd on
the mozilla-aurora repo is significant:

p1(9c37df347485)  5.5s
9c37df347485:    66.3s
this patch:      10.8s

We're still slower than before. But only by ~2x instead of ~12x.

On the tip revisions of layout/base/nsCSSFrameConstructor.cpp file in
the mozilla-unified repo, time went from 12.5s to 14.5s and back to
12.5s. I'm not sure why the mozilla-aurora repo is so slow.

Looking at the code of basefilectx.parents(), there is room for
further improvements. Notably, we still perform redundant calls to
filelog.renamed() and basefilectx._parentfilectx(). However,
introducing a cache for them is not appropriate for the stable branch.
Gregory Szorc - Nov. 5, 2016, 6:22 a.m.
On Fri, Nov 4, 2016 at 11:07 PM, Gregory Szorc <gregory.szorc@gmail.com>
wrote:

> # HG changeset patch
> # User Gregory Szorc <gregory.szorc@gmail.com>
> # Date 1478325927 25200
> #      Fri Nov 04 23:05:27 2016 -0700
> # Branch stable
> # Node ID b0dc7dd87fc8b5efee6e6d8c0bcc6ddfa52d6df3
> # Parent  e5cc44ea12de681d971fcbebb65a7fb71fd1c3c7
> hgweb: cache fctx.parents() in annotate command (issue5414)
>
> 9c37df347485 introduced a call to fctx.parents() for each line in
> annotate output. This function call isn't cheap, as it requires
> linkrev adjustment.
>
> Since multiple lines in annotate output tend to belong to the same
> file revision, a cache of fctx.parents() lookups for each input
> should be effective in the common case. So we implement one.
>
> The effect of this change when requesting /annotate/96ca0ecdcfa/
> browser/locales/en-US/chrome/browser/downloads/downloads.dtd on
> the mozilla-aurora repo is significant:
>
> p1(9c37df347485)  5.5s
> 9c37df347485:    66.3s
> this patch:      10.8s
>
> We're still slower than before. But only by ~2x instead of ~12x.
>
> On the tip revisions of layout/base/nsCSSFrameConstructor.cpp file in
> the mozilla-unified repo, time went from 12.5s to 14.5s and back to
> 12.5s. I'm not sure why the mozilla-aurora repo is so slow.
>
> Looking at the code of basefilectx.parents(), there is room for
> further improvements. Notably, we still perform redundant calls to
> filelog.renamed() and basefilectx._parentfilectx(). However,
> introducing a cache for them is not appropriate for the stable branch.
>

basefilectx.annotate() already performs some linkrev adjustment via
basefilectx.parents(). So I'm pretty sure the basefilectx.parents() calls
in hgweb are redundant with what annotate() is doing. I think there is room
for basefilectx.annotate() to return this data so hgweb doesn't have to
recompute it. But I've never really grokked linkrev adjustment or the
annotate code. If someone else wants to take a shot at this, please do!
Yuya Nishihara - Nov. 5, 2016, 9:27 a.m.
On Fri, 04 Nov 2016 23:07:21 -0700, Gregory Szorc wrote:
> # HG changeset patch
> # User Gregory Szorc <gregory.szorc@gmail.com>
> # Date 1478325927 25200
> #      Fri Nov 04 23:05:27 2016 -0700
> # Branch stable
> # Node ID b0dc7dd87fc8b5efee6e6d8c0bcc6ddfa52d6df3
> # Parent  e5cc44ea12de681d971fcbebb65a7fb71fd1c3c7
> hgweb: cache fctx.parents() in annotate command (issue5414)

Looks good, but one nit.

> --- a/mercurial/hgweb/webcommands.py
> +++ b/mercurial/hgweb/webcommands.py
> @@ -861,12 +861,25 @@ def annotate(web, req, tmpl):
>      f = fctx.path()
>      parity = paritygen(web.stripecount)
>  
> +    # parents() is called once per line and several lines likely belong to
> +    # same revision. So it is worth caching.
> +    # TODO there are still redundant operations within basefilectx.parents()
> +    # that could also be cached.
> +    parentscache = {}
>      def parents(f):
> -        for p in f.parents():
> -            yield {
> -                "node": p.hex(),
> -                "rev": p.rev(),
> -            }
> +        rev = f.rev()
> +        if rev in parentscache:
> +            for entry in parentscache[rev]:
> +                yield entry
> +        else:
> +            parentscache[rev] = []
> +            for p in f.parents():
> +                entry = {
> +                    'node': p.hex(),
> +                    'rev': p.rev(),
> +                }
> +                parentscache[rev].append(entry)
> +                yield entry

If the generator is stopped at the first yield, parentscache would be
incomplete.

Patch

diff --git a/mercurial/hgweb/webcommands.py b/mercurial/hgweb/webcommands.py
--- a/mercurial/hgweb/webcommands.py
+++ b/mercurial/hgweb/webcommands.py
@@ -861,12 +861,25 @@  def annotate(web, req, tmpl):
     f = fctx.path()
     parity = paritygen(web.stripecount)
 
+    # parents() is called once per line and several lines likely belong to
+    # same revision. So it is worth caching.
+    # TODO there are still redundant operations within basefilectx.parents()
+    # that could also be cached.
+    parentscache = {}
     def parents(f):
-        for p in f.parents():
-            yield {
-                "node": p.hex(),
-                "rev": p.rev(),
-            }
+        rev = f.rev()
+        if rev in parentscache:
+            for entry in parentscache[rev]:
+                yield entry
+        else:
+            parentscache[rev] = []
+            for p in f.parents():
+                entry = {
+                    'node': p.hex(),
+                    'rev': p.rev(),
+                }
+                parentscache[rev].append(entry)
+                yield entry
 
     def annotate(**map):
         if util.binary(fctx.data()):