Patchwork [4,of,7,V2] bundle: include advisory rev branch cache part in bundle2 bundle

login
register
mail settings
Submitter Boris Feld
Date March 1, 2018, 8:44 p.m.
Message ID <d0012d0077b4cb19cbd3.1519937059@FB>
Download mbox | patch
Permalink /patch/28589/
State Superseded
Headers show

Comments

Boris Feld - March 1, 2018, 8:44 p.m.
# HG changeset patch
# User Boris Feld <boris.feld@octobus.net>
# Date 1519230780 -3600
#      Wed Feb 21 17:33:00 2018 +0100
# Node ID d0012d0077b4cb19cbd32b3f1b1741b3d53edc6f
# Parent  73e85c62e2927df8a905430436ec95686c28afd6
# EXP-Topic wire-rbc
# Available At https://bitbucket.org/octobus/mercurial-devel/
#              hg pull https://bitbucket.org/octobus/mercurial-devel/ -r d0012d0077b4
bundle: include advisory rev branch cache part in bundle2 bundle

`hg bundle` command producing bundle2 will now include an optional part
containing the revision-branch cache data.

The data sent are mostly nodes so it is quite compact.  The goal of the
rev-branch-cache is to speed up branch map computation, especially when the
branchmap gets invalidated so we send data for all exchanged changesets. In
addition, computing the relevant heads to send in case of partial pulling would
be challenging.

As a reminder, the rev branch cache data significantly speed up branch
computation. Having it around provides a small speedup to pull/clone and much
higher tolerance to branch map cache invalidation that might happens from later
commands.

On the Mercurial repository, computing the visible branchmap from scratch move
from 2.00 seconds to 0.34s (a -83% speedup).

Using this new part, Unbundling the full Mercurial repository moves from 25.736
seconds to 24.030 seconds (around -7% speedup). The bundle size increase is
around 3% (from 22.43 MB to 23.13MB)


On an half a million revision repository with twenty thousand
branches, computing the branchmap moves from 75 seconds to 45 second (-40%) if
the caches is used.

A bundle containing 50 000 changesets in such repository get a 0.5% size
increase from such part for a -3% unbundling time speedup.

Patch

diff --git a/mercurial/bundle2.py b/mercurial/bundle2.py
--- a/mercurial/bundle2.py
+++ b/mercurial/bundle2.py
@@ -1591,6 +1591,7 @@  def _addpartsfromopts(ui, repo, bundler,
         part.addparam('targetphase', '%d' % phases.secret, mandatory=False)
 
     addparttagsfnodescache(repo, bundler, outgoing)
+    addpartrevbranchcache(repo, bundler, outgoing)
 
     if opts.get('obsolescence', False):
         obsmarkers = repo.obsstore.relevantmarkers(outgoing.missing)
diff --git a/tests/test-bundle-phases.t b/tests/test-bundle-phases.t
--- a/tests/test-bundle-phases.t
+++ b/tests/test-bundle-phases.t
@@ -42,6 +42,7 @@  Phases are restored when unbundling
       26805aba1e600a82e93661149f2313866a221a7b
       f585351a92f85104bff7c284233c338b10eb1df7
       9bc730a19041f9ec7cb33c626e811aa233efb18c
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       26805aba1e600a82e93661149f2313866a221a7b draft
   $ hg strip --no-backup C
@@ -233,6 +234,7 @@  Restore bundle of entire repo
       dc0947a82db884575bb76ea10ac97b08536bfa03
       4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
       03ca77807e919db8807c3749086dc36fb478cac0
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       dc0947a82db884575bb76ea10ac97b08536bfa03 public
       03ca77807e919db8807c3749086dc36fb478cac0 draft
@@ -258,6 +260,7 @@  Restore bundle of entire repo
   changegroup -- {nbchanges: 2, targetphase: 2, version: 02}
       112478962961147124edd43549aedd1a335e44bf
       4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
   $ rm bundle
 
@@ -269,6 +272,7 @@  Restore bundle of entire repo
       112478962961147124edd43549aedd1a335e44bf
       dc0947a82db884575bb76ea10ac97b08536bfa03
       4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       dc0947a82db884575bb76ea10ac97b08536bfa03 public
   $ rm bundle
@@ -280,6 +284,7 @@  Restore bundle of entire repo
   changegroup -- {nbchanges: 2, targetphase: 2, version: 02}
       4e4f9194f9f181c57f62e823e8bdfa46ab9e4ff4
       03ca77807e919db8807c3749086dc36fb478cac0
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       03ca77807e919db8807c3749086dc36fb478cac0 draft
   $ rm bundle
diff --git a/tests/test-bundle-type.t b/tests/test-bundle-type.t
--- a/tests/test-bundle-type.t
+++ b/tests/test-bundle-type.t
@@ -76,6 +76,7 @@  test bundle types
   Stream params: {}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   none-v2
   
   % test bundle type bzip2
@@ -85,6 +86,7 @@  test bundle types
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   bzip2-v2
   
   % test bundle type gzip
@@ -94,6 +96,7 @@  test bundle types
   Stream params: {Compression: GZ}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   gzip-v2
   
   % test bundle type none-v2
@@ -103,6 +106,7 @@  test bundle types
   Stream params: {}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   none-v2
   
   % test bundle type v2
@@ -112,6 +116,7 @@  test bundle types
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   bzip2-v2
   
   % test bundle type v1
@@ -150,12 +155,12 @@  Compression level can be adjusted for bu
   $ hg bundle -a -t gzip-v2 gzip-v2.hg
   1 changesets found
   $ f --size gzip-v2.hg
-  gzip-v2.hg: size=427
+  gzip-v2.hg: size=468
 
   $ hg --config experimental.bundlecomplevel=1 bundle -a -t gzip-v2 gzip-v2-level1.hg
   1 changesets found
   $ f --size gzip-v2-level1.hg
-  gzip-v2-level1.hg: size=435
+  gzip-v2-level1.hg: size=475
 
   $ cd ..
 
@@ -171,6 +176,7 @@  Compression level can be adjusted for bu
   Stream params: {Compression: ZS}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   zstd-v2
   
   % test bundle type zstd-v2
@@ -180,6 +186,7 @@  Compression level can be adjusted for bu
   Stream params: {Compression: ZS}
   changegroup -- {nbchanges: 1, version: 02}
       c35a0f9217e65d1fdb90c936ffa7dbe679f83ddf
+  cache:rev-branch-cache -- {}
   zstd-v2
   
 
diff --git a/tests/test-bundle.t b/tests/test-bundle.t
--- a/tests/test-bundle.t
+++ b/tests/test-bundle.t
@@ -774,7 +774,7 @@  bundle single branch
   list of changesets:
   1a38c1b849e8b70c756d2d80b0b9a3ac0b7ea11a
   057f4db07f61970e1c11e83be79e9d08adc4dc31
-  bundle2-output-bundle: "HG20", (1 params) 1 parts total
+  bundle2-output-bundle: "HG20", (1 params) 2 parts total
   bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
   bundling: 1/2 changesets (50.00%)
   bundling: 2/2 changesets (100.00%)
@@ -783,6 +783,7 @@  bundle single branch
   bundling: b 1/3 files (33.33%)
   bundling: b1 2/3 files (66.67%)
   bundling: x 3/3 files (100.00%)
+  bundle2-output-part: "cache:rev-branch-cache" streamed payload
 
 == Test for issue3441
 
diff --git a/tests/test-clonebundles.t b/tests/test-clonebundles.t
--- a/tests/test-clonebundles.t
+++ b/tests/test-clonebundles.t
@@ -163,7 +163,7 @@  changes, clone bundles produced by new M
 by old clients.
 
   $ f --size --hexdump full.hg
-  full.hg: size=396
+  full.hg: size=442
   0000: 48 47 32 30 00 00 00 0e 43 6f 6d 70 72 65 73 73 |HG20....Compress|
   0010: 69 6f 6e 3d 47 5a 78 9c 63 60 60 d0 e4 76 f6 70 |ion=GZx.c``..v.p|
   0020: f4 73 77 75 0f f2 0f 0d 60 00 02 46 46 76 26 4e |.swu....`..FFv&N|
@@ -188,7 +188,10 @@  by old clients.
   0150: 88 75 34 36 75 04 82 55 17 14 36 a4 38 10 04 d8 |.u46u..U..6.8...|
   0160: 21 01 9a b1 83 f7 e9 45 8b d2 56 c7 a3 1f 82 52 |!......E..V....R|
   0170: d7 8a 78 ed fc d5 76 f1 36 25 81 89 c7 ad ec 90 |..x...v.6%......|
-  0180: 54 47 75 2b 89 49 b1 00 d2 8a eb 92             |TGu+.I......|
+  0180: 54 47 75 2b 89 48 b1 b2 62 ce 8e ce 1e ae 56 41 |TGu+.H..b.....VA|
+  0190: ae 61 ba 4e 41 8e 7e ce 1e ba 60 01 a0 14 23 58 |.a.NA.~...`...#X|
+  01a0: 81 35 c8 7d 40 cc 04 e2 a4 a4 a6 25 96 e6 94 60 |.5.}@......%...`|
+  01b0: 33 17 5f 54 00 00 01 1b 0a ec                   |3._T......|
 
   $ echo "http://localhost:$HGPORT1/full.hg" > server/.hg/clonebundles.manifest
   $ hg clone -U http://localhost:$HGPORT full-bundle
@@ -529,14 +532,14 @@  Test clone bundle retrieved through bund
   $ cat hg.pid >> $DAEMON_PIDS
 
   $ hg -R server debuglfput gz-a.hg
-  f6eca29e25359f6a92f1ea64527cdcf1b5abe62a
+  14ee2f0b3f1d14aeeb2fe037e09fc295c3cf59f5
 
   $ cat > server/.hg/clonebundles.manifest << EOF
-  > largefile://f6eca29e25359f6a92f1ea64527cdcf1b5abe62a BUNDLESPEC=gzip-v2
+  > largefile://14ee2f0b3f1d14aeeb2fe037e09fc295c3cf59f5 BUNDLESPEC=gzip-v2
   > EOF
 
   $ hg clone -U http://localhost:$HGPORT largefile-provided --traceback
-  applying clone bundle from largefile://f6eca29e25359f6a92f1ea64527cdcf1b5abe62a
+  applying clone bundle from largefile://14ee2f0b3f1d14aeeb2fe037e09fc295c3cf59f5
   adding changesets
   adding manifests
   adding file changes
diff --git a/tests/test-debugbundle.t b/tests/test-debugbundle.t
--- a/tests/test-debugbundle.t
+++ b/tests/test-debugbundle.t
@@ -34,6 +34,7 @@  Terse output:
   changegroup -- {nbchanges: 2, version: 02}
       0e067c57feba1a5694ca4844f05588bb1bf82342
       991a3460af53952d10ec8a295d3d2cc2e5fa9690
+  cache:rev-branch-cache -- {}
 
 Verbose output:
 
@@ -72,5 +73,6 @@  Verbose output:
   
       c
       b80de5d138758541c5f05265ad144ab9fa86d1db 0000000000000000000000000000000000000000 0000000000000000000000000000000000000000 991a3460af53952d10ec8a295d3d2cc2e5fa9690 0000000000000000000000000000000000000000 0
+  cache:rev-branch-cache -- {}
 
   $ cd ..
diff --git a/tests/test-generaldelta.t b/tests/test-generaldelta.t
--- a/tests/test-generaldelta.t
+++ b/tests/test-generaldelta.t
@@ -159,6 +159,7 @@  Test that strip bundle use bundle2
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       1c5d4dc9a8b8d6e1750966d343e94db665e7a1e9
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       1c5d4dc9a8b8d6e1750966d343e94db665e7a1e9 draft
 
diff --git a/tests/test-obsolete-changeset-exchange.t b/tests/test-obsolete-changeset-exchange.t
--- a/tests/test-obsolete-changeset-exchange.t
+++ b/tests/test-obsolete-changeset-exchange.t
@@ -95,10 +95,12 @@  check-that bundle can contain markers:
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       f89bcc95eba5174b1ccc3e33a82e84c96e8338ee
+  cache:rev-branch-cache -- {}
   $ hg debugbundle ../f89bcc95eba5-obs.hg
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       f89bcc95eba5174b1ccc3e33a82e84c96e8338ee
+  cache:rev-branch-cache -- {}
   obsmarkers -- {}
       version: 1 (70 bytes)
       9d73aac1b2ed7d53835eaeec212ed41ea47da53a f89bcc95eba5174b1ccc3e33a82e84c96e8338ee 0 (Thu Jan 01 00:00:00 1970 +0000) {'user': 'test'}
diff --git a/tests/test-obsolete.t b/tests/test-obsolete.t
--- a/tests/test-obsolete.t
+++ b/tests/test-obsolete.t
@@ -1411,6 +1411,7 @@  Test ability to pull changeset with loca
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       e008cf2834908e5d6b0f792a9d4b0e2272260fb8
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       e008cf2834908e5d6b0f792a9d4b0e2272260fb8 draft
 
@@ -1452,6 +1453,7 @@  Testing that strip remove markers:
   changegroup -- {nbchanges: 2, version: 02}
       e016b03fd86fcccc54817d120b90b751aaf367d6
       b0551702f918510f01ae838ab03a463054c67b46
+  cache:rev-branch-cache -- {}
   obsmarkers -- {}
       version: 1 (92 bytes)
       e008cf2834908e5d6b0f792a9d4b0e2272260fb8 b0551702f918510f01ae838ab03a463054c67b46 0 (Thu Jan 01 00:00:00 1970 +0000) {'ef1': '8', 'operation': 'amend', 'user': 'test'}
diff --git a/tests/test-patchbomb.t b/tests/test-patchbomb.t
--- a/tests/test-patchbomb.t
+++ b/tests/test-patchbomb.t
@@ -390,14 +390,15 @@  test bundle and description:
   Content-Disposition: attachment; filename="bundle.hg"
   Content-Transfer-Encoding: base64
   
-  SEcyMAAAAA5Db21wcmVzc2lvbj1CWkJaaDkxQVkmU1nYvy2xAAAJf//7vFYQXD1/4H7R09C/470I
-  Ak0E4pe4SIIIgQSgGEQOcLAA5VBKqeppMxTI0YjQNBgQMQDI0GgMhtR6I0GI2p6I0yeSEVT9MiYn
-  qjCYQwCBtARptARgBNDEwAGiDCMA40NGjQaNA0AAAAADIAAAA0BkABktCk6qObVxZ2A/33KHibLr
-  UQ4BwkgcPcmuCUAQZCztIWgR1SpBS6IqbIij4UFwhnnMkElcFTqoucIWbsBPK3l+6c+xYaVBWsJo
-  aT0OV/YAOvLrziifDQMJOMIaaYce9agtI2EwQBAq089UiRU+evFHSLRBT7Wa/D/YBaUtU5ezvtr3
-  6yrIS4Iyp9VWESdWPEi6VjRjdcEY4HvbmDIVEAEVJIUrHNIBx/MmnBBRkw8tSlCQ8ABZxf5ejgBI
-  pP5TSQPLVMYbq1qbBPmWN0LYVlAvRbP4X512kDQZ9y4TQbvoZmhe+54sRsEJ8GW3hMJjERh0NNlg
-  aB+3Cw/4u5IpwoSGxfltiA==
+  SEcyMAAAAA5Db21wcmVzc2lvbj1CWkJaaDkxQVkmU1kHdO0GAAAN////vFcSXL9/8H7R09C/578I
+  Ak0E4pe4SIIIgQSgGEQOcLABGYYNKiaaZGEyYjJhGTTRpiHogxGmTRiGRkNMIwhhPSbQJtpQiJkn
+  poyk9I0PUeoNNNBkeUAHqGgD0Ro0NNBoBoaMagNBoNCpNPUemp6QGmgyaPSGmQxGhkZDQbUaBkHp
+  MhoaANNMhkIyIauvSJPL4aUXjIQemQXkoaqOKqAQDIABsZALnf0yCLAyvmktzDWBCVHO6bb6kCqE
+  ZobVEhmMBjs0oQzekgs6PgZSyII8zy9mmG9To49ZlN6TaX5BxlS7cJiuICUdyjNQPIIdQs1Qqqqk
+  JZ2/BksYcU4HQyssZcpkoMco6gRc888KF9BO7BvuSuIPz7A4crBoaQB+euFU1ilz8yIBBmNBDgRX
+  pVh4zkmPiSKcqRJxcshMqh0vkKlgQDTcOujtdmnMVBZfQiPPemcHm2098VJyHBAOqOwluyIKyG92
+  JAR0CCu9SB5q9DyPHUdc5yB5CurIZHt3GM0dCiQRIN0EAcQNmTYTiHdi6B6Dc/ma0hrmSCQXBzfU
+  BEwthEg0YGaJf4u5IpwoSAO6doMA
   --===============*==-- (glob)
 
 with a specific bundle type
diff --git a/tests/test-rebase-conflicts.t b/tests/test-rebase-conflicts.t
--- a/tests/test-rebase-conflicts.t
+++ b/tests/test-rebase-conflicts.t
@@ -297,8 +297,9 @@  Check that the right ancestors is used w
   list of changesets:
   e31216eec445e44352c5f01588856059466a24c9
   2f2496ddf49d69b5ef23ad8cf9fb2e0e4faf0ac2
-  bundle2-output-bundle: "HG20", (1 params) 2 parts total
+  bundle2-output-bundle: "HG20", (1 params) 3 parts total
   bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
+  bundle2-output-part: "cache:rev-branch-cache" streamed payload
   bundle2-output-part: "phase-heads" 24 bytes payload
   saved backup bundle to $TESTTMP/issue4041/.hg/strip-backup/e31216eec445-15f7a814-rebase.hg
   3 changesets found
@@ -306,8 +307,9 @@  Check that the right ancestors is used w
   4c9fbe56a16f30c0d5dcc40ec1a97bbe3325209c
   19c888675e133ab5dff84516926a65672eaf04d9
   2a7f09cac94c7f4b73ebd5cd1a62d3b2e8e336bf
-  bundle2-output-bundle: "HG20", 2 parts total
+  bundle2-output-bundle: "HG20", 3 parts total
   bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
+  bundle2-output-part: "cache:rev-branch-cache" streamed payload
   bundle2-output-part: "phase-heads" 24 bytes payload
   adding branch
   bundle2-input-bundle: with-transaction
@@ -321,12 +323,15 @@  Check that the right ancestors is used w
   adding f1.txt revisions
   added 2 changesets with 2 changes to 1 files
   bundle2-input-part: total payload size 1686
+  bundle2-input-part: "cache:rev-branch-cache" supported
+  bundle2-input-part: total payload size 74
   bundle2-input-part: "phase-heads" supported
   bundle2-input-part: total payload size 24
-  bundle2-input-bundle: 1 parts total
+  bundle2-input-bundle: 2 parts total
   updating the branch cache
   invalid branchheads cache (served): tip differs
   rebase completed
+  truncating cache/rbc-revs-v1 to 0
 
 Test minimization of merge conflicts
   $ hg up -q null
diff --git a/tests/test-strip.t b/tests/test-strip.t
--- a/tests/test-strip.t
+++ b/tests/test-strip.t
@@ -214,6 +214,7 @@ 
   Stream params: {Compression: BZ}
   changegroup -- {nbchanges: 1, version: 02}
       264128213d290d868c54642d13aeaa3675551a78
+  cache:rev-branch-cache -- {}
   phase-heads -- {}
       264128213d290d868c54642d13aeaa3675551a78 draft
   $ hg pull .hg/strip-backup/*
@@ -843,13 +844,13 @@  check strip behavior
   list of changesets:
   6625a516847449b6f0fa3737b9ba56e9f0f3032c
   d8db9d1372214336d2b5570f20ee468d2c72fa8b
-  bundle2-output-bundle: "HG20", (1 params) 2 parts total
+  bundle2-output-bundle: "HG20", (1 params) 3 parts total
   bundle2-output-part: "changegroup" (params: 1 mandatory 1 advisory) streamed payload
+  bundle2-output-part: "cache:rev-branch-cache" streamed payload
   bundle2-output-part: "phase-heads" 24 bytes payload
   saved backup bundle to $TESTTMP/issue4736/.hg/strip-backup/6625a5168474-345bb43d-backup.hg
   updating the branch cache
   invalid branchheads cache (served): tip differs
-  truncating cache/rbc-revs-v1 to 24
   $ hg log -G
   o  changeset:   2:5c51d8d6557d
   |  tag:         tip
diff --git a/tests/test-tags.t b/tests/test-tags.t
--- a/tests/test-tags.t
+++ b/tests/test-tags.t
@@ -729,6 +729,7 @@  Check that the bundle includes cache dat
       f63cc8fe54e4d326f8d692805d70e092f851ddb1
       40f0358cb314c824a5929ee527308d90e023bc10
   hgtagsfnodes -- {}
+  cache:rev-branch-cache -- {}
 
 Check that local clone includes cache data