Patchwork [3,of,4,V2] streamclone: clear caches after writing changes into files for visibility

login
register
mail settings
Submitter Katsunori FUJIWARA
Date Sept. 11, 2016, 6:11 p.m.
Message ID <0f6ba83a54fab382452f.1473617515@feefifofum>
Download mbox | patch
Permalink /patch/16589/
State Accepted
Headers show

Comments

Katsunori FUJIWARA - Sept. 11, 2016, 6:11 p.m.
# HG changeset patch
# User FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
# Date 1473617189 -32400
#      Mon Sep 12 03:06:29 2016 +0900
# Node ID 0f6ba83a54fab382452faf389265ef636d0c42a1
# Parent  8f9b06ec1b6dbca4c3307dfedb23af46106b33a0
streamclone: clear caches after writing changes into files for visibility

Before this patch, streamclone-ed changes are invisible via @filecache
properties to in-process procedures before closing transaction
(e.g. pretxnclose python hook), if corresponded property is cached
before consumev1(). Strictly speaking, caching should occur inside
(store) lock for transaction.

repo.invalidate() after closing transaction is too late to force
@filecache properties to be reloaded from changed files at next
access.

For visibility of streamclone-ed changes to in-process procedures
before closing transaction, this patch clears caches just after
writing changes into files.

BTW, regardless of changing in this patch, clearing cached properties
in consumev1() causes inconsistency, if (1) transaction is started and
(2) any @filecache property is changed before consumev1().

This patch also adds the comment to fix this (potential) inconsistency
in the future.

Patch

diff --git a/mercurial/streamclone.py b/mercurial/streamclone.py
--- a/mercurial/streamclone.py
+++ b/mercurial/streamclone.py
@@ -299,6 +299,20 @@  def consumev1(repo, fp, filecount, bytec
         repo.ui.progress(_('clone'), 0, total=bytecount, unit=_('bytes'))
         start = time.time()
 
+        # TODO: get rid of (potential) inconsistency
+        #
+        # If transaction is started and any @filecache property is
+        # changed at this point, it causes inconsistency between
+        # in-memory cached property and streamclone-ed file on the
+        # disk. Nested transaction prevents transaction scope "clone"
+        # below from writing in-memory changes out at the end of it,
+        # even though in-memory changes are discarded at the end of it
+        # regardless of transaction nesting.
+        #
+        # But transaction nesting can't be simply prohibited, because
+        # nesting occurs also in ordinary case (e.g. enabling
+        # clonebundles).
+
         with repo.transaction('clone'):
             with repo.svfs.backgroundclosing(repo.ui, expectedcount=filecount):
                 for i in xrange(filecount):
@@ -322,8 +336,9 @@  def consumev1(repo, fp, filecount, bytec
                                              total=bytecount, unit=_('bytes'))
                             ofp.write(chunk)
 
-        # Writing straight to files circumvented the inmemory caches
-        repo.invalidate(clearfilecache=True)
+            # force @filecache properties to be reloaded from
+            # streamclone-ed file at next access
+            repo.invalidate(clearfilecache=True)
 
         elapsed = time.time() - start
         if elapsed <= 0:
diff --git a/tests/test-bundle.t b/tests/test-bundle.t
--- a/tests/test-bundle.t
+++ b/tests/test-bundle.t
@@ -310,9 +310,51 @@  Unpacking packed1 bundles with "hg unbun
 
 packed1 can be consumed from debug command
 
+(this also confirms that streamclone-ed changes are visible via
+@filecache properties to in-process procedures before closing
+transaction)
+
+  $ cat > $TESTTMP/showtip.py <<EOF
+  > from __future__ import absolute_import
+  > 
+  > def showtip(ui, repo, hooktype, **kwargs):
+  >     ui.warn('%s: %s\n' % (hooktype, repo['tip'].hex()[:12]))
+  > 
+  > def reposetup(ui, repo):
+  >     # this confirms (and ensures) that (empty) 00changelog.i
+  >     # before streamclone is already cached as repo.changelog
+  >     ui.setconfig('hooks', 'pretxnopen.showtip', showtip)
+  > 
+  >     # this confirms that streamclone-ed changes are visible to
+  >     # in-process procedures before closing transaction
+  >     ui.setconfig('hooks', 'pretxnclose.showtip', showtip)
+  > 
+  >     # this confirms that streamclone-ed changes are still visible
+  >     # after closing transaction
+  >     ui.setconfig('hooks', 'txnclose.showtip', showtip)
+  > EOF
+  $ cat >> $HGRCPATH <<EOF
+  > [extensions]
+  > showtip = $TESTTMP/showtip.py
+  > EOF
+
   $ hg -R packed debugapplystreamclonebundle packed.hg
   6 files to transfer, 2.60 KB of data
+  pretxnopen: 000000000000
+  pretxnclose: aa35859c02ea
   transferred 2.60 KB in *.* seconds (* */sec) (glob)
+  txnclose: aa35859c02ea
+
+(for safety, confirm visibility of streamclone-ed changes by another
+process, too)
+
+  $ hg -R packed tip -T "{node|short}\n"
+  aa35859c02ea
+
+  $ cat >> $HGRCPATH <<EOF
+  > [extensions]
+  > showtip = !
+  > EOF
 
 Does not work on non-empty repo