Patchwork revset: speed up '_matchfiles'

login
register
mail settings
Submitter Pierre-Yves David
Date Nov. 19, 2015, 9:20 a.m.
Message ID <f26c052b5e2b48228309.1447924848@marginatus.alto.octopoid.net>
Download mbox | patch
Permalink /patch/11507/
State Accepted
Headers show

Comments

Pierre-Yves David - Nov. 19, 2015, 9:20 a.m.
# HG changeset patch
# User Pierre-Yves David <pierre-yves.david@fb.com>
# Date 1447917783 28800
#      Wed Nov 18 23:23:03 2015 -0800
# Node ID f26c052b5e2b48228309fc36a5a8f4574a1a84a0
# Parent  53c668dc6b1601c8bda5098df4e33030dbdb454d
# EXP-Topic speedupmatches
# Available At http://hg.netv6.net/marmoute-wip/mercurial/
#              hg pull http://hg.netv6.net/marmoute-wip/mercurial/ -r f26c052b5e2b
revset: speed up '_matchfiles'

File matching is done by applying the matcher to all elements in the 'file'
field of all changesets in the repository. This requires to read/parse all
changesets in the repository and do a lot of matching. However about 1/3 of the
time of the function is used to create 'changectx' object and retrieve their
'file' field.

This is far too much overhead so we are skipping the changectx layer and
directly access the data from the changelog. This provide use significant speed
up:

repository: mozilla central 252524 revisions
command: hg perfrevset '_matchfiles("p:browser")'
Before: 15.899687s
After:  10.011705s

Slowdown is even more significant if you have a lot of namespace that slowdown
lookup.

The time is now spent with this approximate repartition:

  Matcher: 20%
    regexp matching: 10%
  changelog.read: 80%
    reading revision: 60%
      checking hash: 15%
      decompression: 15%
      reading chunk: 30%
    changelog parsing: 20%
      decoding to local: 10%

The next easy win is probably to have more of the changelog stack implemented
using the CPython api.
Yuya Nishihara - Nov. 19, 2015, 1:59 p.m.
On Thu, 19 Nov 2015 01:20:48 -0800, Pierre-Yves David wrote:
> # HG changeset patch
> # User Pierre-Yves David <pierre-yves.david@fb.com>
> # Date 1447917783 28800
> #      Wed Nov 18 23:23:03 2015 -0800
> # Node ID f26c052b5e2b48228309fc36a5a8f4574a1a84a0
> # Parent  53c668dc6b1601c8bda5098df4e33030dbdb454d
> # EXP-Topic speedupmatches
> # Available At http://hg.netv6.net/marmoute-wip/mercurial/
> #              hg pull http://hg.netv6.net/marmoute-wip/mercurial/ -r f26c052b5e2b
> revset: speed up '_matchfiles'
> 
> File matching is done by applying the matcher to all elements in the 'file'
> field of all changesets in the repository. This requires to read/parse all
> changesets in the repository and do a lot of matching. However about 1/3 of the
> time of the function is used to create 'changectx' object and retrieve their
> 'file' field.
> 
> This is far too much overhead so we are skipping the changectx layer and
> directly access the data from the changelog. This provide use significant speed
> up:
> 
> repository: mozilla central 252524 revisions
> command: hg perfrevset '_matchfiles("p:browser")'
> Before: 15.899687s
> After:  10.011705s

Nice, pushed to the clowncopter. Thanks.

Patch

diff --git a/mercurial/revset.py b/mercurial/revset.py
--- a/mercurial/revset.py
+++ b/mercurial/revset.py
@@ -1162,12 +1162,20 @@  def _matchfiles(repo, subset, x):
         default = 'glob'
 
     m = matchmod.match(repo.root, repo.getcwd(), pats, include=inc,
                        exclude=exc, ctx=repo[rev], default=default)
 
+    # This directly read the changelog data as creating changectx for all
+    # revisions is quite expensive.
+    getchangeset = repo.changelog.read
+    wdirrev = node.wdirrev
     def matches(x):
-        for f in repo[x].files():
+        if x == wdirrev:
+            files = repo[x].files()
+        else:
+            files = getchangeset(x)[3]
+        for f in files:
             if m(f):
                 return True
         return False
 
     return subset.filter(matches)