From patchwork Wed May 28 06:00:47 2014 Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: [3,of,3,RFC] localrepository: add caches to transaction (issue4255) From: Gregory Szorc X-Patchwork-Id: 4882 Message-Id: <12c2a7ae1d443e6ea89e.1401256847@vm-ubuntu-main.gateway.sonic.net> To: mercurial-devel@selenic.com Date: Tue, 27 May 2014 23:00:47 -0700 # HG changeset patch # User Gregory Szorc # Date 1401253147 25200 # Tue May 27 21:59:07 2014 -0700 # Node ID 12c2a7ae1d443e6ea89ea6f8f1032bf2178af15d # Parent 02939296b8052833d4c8bfa6e2e2f1371fe5c517 localrepository: add caches to transaction (issue4255) localrepository.transaction() now creates a chained transaction bound to the .hg opener in addition to the .hg/store opener. This new transaction creates backups of most files under .hg/cache. With this change, transaction rollbacks will restore caches, thus preventing cache invalidation and recomputation. This will result in considerable performance wins on repositories where cache generation is slow. On a repository with 12,421 heads and manifests with near 100,000 entries, branch caches can take ~17 minutes to generate (259s for base, 68s for immutable, 723s for served). On a server that had hooks causing branch cache updates, transaction rollbacks resulted in branch cache invalidation and an expensive computation. This patch makes the problem go away. THIS PATCH CURRENTLY CAUSES A REGRESSION IN test-fncache.t. THIS PATCH SHOULD ALSO HAVE EXPLICIT TEST COVERAGE THAT CACHES ARE RESTORED. diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py --- a/mercurial/localrepo.py +++ b/mercurial/localrepo.py @@ -848,8 +848,16 @@ class localrepository(object): def wwritedata(self, filename, data): return self._filter(self._decodefilterpats, filename, data) + def _cachetransactionblacklist(self): + """Set of cache files that are not safe to include in transactions. + + Files in .hg/cache/ are included in transactions by default. To + prevent that from happening, add an entry to this set. + """ + return set(['storehash']) + def transaction(self, desc, report=None): tr = self._transref and self._transref() or None if tr and tr.running(): return tr.nest() @@ -870,8 +878,27 @@ class localrepository(object): "journal", aftertrans(renames), self.store.createmode, onclose) + + # Add caches to a chained transaction. + if self.ui.configbool("format", "usestore", True): + try: + def noreport(msg): + pass + tr2 = transaction.transaction(noreport, self.opener, + "journal", + createmode=self.store.createmode) + blacklist = self._cachetransactionblacklist() + for filename, mode in self.vfs.readdir("cache"): + if filename not in blacklist: + tr2.addbackup("cache/%s" % filename, hardlink=False) + tr.chaintransaction(tr2) + + except OSError, e: + if e.errno != errno.ENOENT: + raise + self._transref = weakref.ref(tr) return tr def _journalfiles(self):