Patchwork D3532: wireprotov2: define and implement "rawstorefile" command

login
register
mail settings
Submitter phabricator
Date May 11, 2018, 10:35 p.m.
Message ID <differential-rev-PHID-DREV-hmxoghpyyt6jqx6tfrnn-req@phab.mercurial-scm.org>
Download mbox | patch
Permalink /patch/31520/
State Superseded
Headers show

Comments

phabricator - May 11, 2018, 10:35 p.m.
indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

REVISION SUMMARY
  stream_out - the previous command for sending raw revlog files -
  was not carried forward to protocol version 2.
  
  This commit introduces a minimal viable replacement for stream_out
  in wire protocol version 2.
  
  The new command allows obtaining "raw store files" - essentially
  files from Mercurial's store as they exist on disk.
  
  The command currently only allows obtaining the changelog or the
  changelog plus root manifestlog. This is the feature set required
  to support partial clones where only files data is partial.
  
  We'll probably want to implement support for retrieving changelog
  and manifest data via dedicated commands in order to facilitate
  partial clone. And if we do decide to keep a command for streaming
  "raw" files, we'll want to support tree manifests. This command
  is very much a minimum viable implementation. I foresee things
  changing substantially. None of wire protocol version 2 is
  covered by BC yet. So hopefully the barrier to entry is low.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3532

AFFECTED FILES
  mercurial/help/internals/wireprotocol.txt
  mercurial/wireprotov2server.py
  tests/test-http-protocol.t
  tests/test-wireproto-command-capabilities.t
  tests/test-wireproto-command-rawstorefile.t

CHANGE DETAILS




To: indygreg, #hg-reviewers
Cc: mercurial-devel
phabricator - May 12, 2018, 12:21 a.m.
martinvonz added inline comments.

INLINE COMMENTS

> wireprotov2server.py:548-549
> +
> +    with repo.lock():
> +        topfiles = list(repo.store.topfiles())
> +

I understand that you don't want to lock the repo for the entire operation, but I assume that also means that the result may fail `hg verify`? If changelog is always sent before the manifest (is it?), then you might have some orphan entries in the manifest if it had been written while the changelog was read.

> wireprotov2server.py:554
> +
> +    for name, encodedname, size in topfiles:
> +        if what.startswith(b'changelog') and name.startswith(b'00changelog'):

tree manifest support might be as easy as also iterating over data files, stripping leading 'meta/' and trailing '00manifest.[id]' and testing against repo.narrowmatch().visitdir()

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3532

To: indygreg, #hg-reviewers
Cc: martinvonz, mercurial-devel
phabricator - May 12, 2018, 5:36 p.m.
joerg.sonnenberger added a comment.


  Similar to Martin's question, I would like to allow streaming clones without any locks. For that to work, one party needs to know how to truncate additional undesired data. That can be fully done by the client or the client could send a size or revision hint to the server.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3532

To: indygreg, #hg-reviewers
Cc: joerg.sonnenberger, martinvonz, mercurial-devel
phabricator - May 12, 2018, 6:13 p.m.
indygreg added a comment.


  In order to support streaming clone without any locks *and* for the result of that clone to pass `hg verify` with no warnings about unreferenced revisions (assuming the server was clean to begin with), I believe we would need to scan the changelog for all referenced manifest nodes and then find the end offset of the last node in the manifest. We would then send the manifest up to that offset.
  
  Then for filelogs, we would need to do something similar for every file.
  
  This doesn't scale.
  
  I see the following solutions to this problem:
  
  1. Obtaining a lock, determining file sizes, and only streaming files up to their known sizes. (This is the current solution.)
  2. Do not obtain a lock, send all manifest and filelog data. This potentially results the client receiving extra manifest and file revisions if the server was in the middle of a transaction when obtaining data. Also, there are race conditions involving a rollback that we'd need to worry about.
  3. Tracking the offsets of all files up to the last transaction so they can be obtained with a lock.
  
  Anyway, I'm not a super big fan of *stream clones*. Their existence is a glorified hack to make clones faster. Their existence is a massive layering violation because it makes the server's storage implementation the client's. This *can* be useful. But I'd rather focus on making normal [partial] clones fast.
  
  I'm considering dropping this command from the series and implementing commands to obtain changeset and manifest data. Then we could implement partial clones without *stream clones* and avoid this debate :)

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3532

To: indygreg, #hg-reviewers
Cc: joerg.sonnenberger, martinvonz, mercurial-devel
phabricator - May 14, 2018, 8:56 a.m.
lothiraldan added a comment.


  First, thank you for your work on the new wire protocol.
  
  We used to send cache file also in streaming clone with the V1 wire protocol. Do you think we would use the new `rawstorefile` command for this purpose? If so, the command name might be confusing as cache files reside outside the store directory.
  
  You were suggesting dropping support for stream clones. No matter its design, do you have ideas how to be as fast as stream clones? Stream clones make cloning huge repositories hours faster than traditional clones.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3532

To: indygreg, #hg-reviewers
Cc: lothiraldan, joerg.sonnenberger, martinvonz, mercurial-devel
phabricator - May 14, 2018, 4:01 p.m.
indygreg added a comment.


  I'm not yet sure what will be done with stream clones. There's a good chance the existing approach more or less gets carried forward. I do concede that it is pretty optimal and we'll have a hard time reproducing its performance.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3532

To: indygreg, #hg-reviewers
Cc: lothiraldan, joerg.sonnenberger, martinvonz, mercurial-devel
phabricator - May 31, 2018, 7:39 p.m.
durin42 added a comment.


  I'm...not thrilled by the abstraction leak in this, but as long as it's strictly temporary on the path to saner partial clones I can live with it.

INLINE COMMENTS

> wireprotocol.txt:1935
> +   (unsigned integer) Number of files being included in the payload.
> +totalsize
> +   (unsigned integer) Total size of all transferred file data, in bytes.

Having this here makes it infeasible to generate the revlog files on the fly, which means non-traditional storage backends won't ever be able to implement this endpoint efficiently.

Do we care? Should we note that this method is going to go away in all likelihood?

> martinvonz wrote in wireprotov2server.py:548-549
> I understand that you don't want to lock the repo for the entire operation, but I assume that also means that the result may fail `hg verify`? If changelog is always sent before the manifest (is it?), then you might have some orphan entries in the manifest if it had been written while the changelog was read.

By grabbing the size along with the list of files inside the lock, that won't happen. :)

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3532

To: indygreg, #hg-reviewers
Cc: durin42, lothiraldan, joerg.sonnenberger, martinvonz, mercurial-devel
phabricator - Aug. 21, 2018, 8:23 p.m.
indygreg abandoned this revision.
indygreg added a comment.


  I'll be taking a different approach to partial clone and the wire protocol in an upcoming series. These patches may get revived someday. But not now.

REPOSITORY
  rHG Mercurial

REVISION DETAIL
  https://phab.mercurial-scm.org/D3532

To: indygreg, #hg-reviewers
Cc: durin42, lothiraldan, joerg.sonnenberger, martinvonz, mercurial-devel

Patch

diff --git a/tests/test-wireproto-command-rawstorefile.t b/tests/test-wireproto-command-rawstorefile.t
new file mode 100644
--- /dev/null
+++ b/tests/test-wireproto-command-rawstorefile.t
@@ -0,0 +1,205 @@ 
+  $ . $TESTDIR/wireprotohelpers.sh
+
+  $ hg init server
+  $ enablehttpv2 server
+
+  $ cd server
+
+  $ echo a0 > a
+  $ echo b0 > b
+  $ echo c0 > c
+  $ mkdir dir0
+  $ echo d0 > dir0/d
+  $ echo e1 > dir0/e
+  $ hg -q commit -A -m initial
+
+  $ echo a1 > a
+  $ hg commit -m 'commit 2'
+  $ echo a2 > a
+  $ hg commit -m 'commit 3'
+  $ hg up -r 0
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ echo a1-branches > a
+  $ hg commit -m 'commit 4 (new head)'
+  created new head
+
+  $ hg up 3
+  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  $ hg mv c c-moved
+  $ hg commit -m 'commit 5 (moved c)'
+
+  $ cd ..
+
+  $ hg -R server serve -p $HGPORT -d --pid-file hg.pid -E error.log
+  $ cat hg.pid > $DAEMON_PIDS
+
+Tests for the "rawstorefile" command
+
+Invalid "what" value results in error
+
+  $ sendhttpv2peer << EOF
+  > command rawstorefile
+  >     what badvalue
+  > EOF
+  creating http peer for wire protocol version 2
+  sending rawstorefile command
+  s>     POST /api/exp-http-v2-0001/ro/rawstorefile HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     accept: application/mercurial-exp-framing-0005\r\n
+  s>     content-type: application/mercurial-exp-framing-0005\r\n
+  s>     content-length: 47\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     user-agent: Mercurial debugwireproto\r\n
+  s>     \r\n
+  s>     \'\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa1DwhatHbadvalueDnameLrawstorefile
+  s> makefile('rb', None)
+  s>     HTTP/1.1 200 OK\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Type: application/mercurial-exp-framing-0005\r\n
+  s>     Transfer-Encoding: chunked\r\n
+  s>     \r\n
+  s>     90\r\n
+  s>     \x88\x00\x00\x01\x00\x02\x012
+  s>     \xa2Eerror\xa2Dargs\x81HbadvalueGmessageXZillegal value for "what" argument (%s): must be "changelog" or "changelog+rootmanifestlog"FstatusEerror
+  s>     \r\n
+  received frame(size=136; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=eos)
+  s>     0\r\n
+  s>     \r\n
+  response: [{b'error': {b'args': [b'badvalue'], b'message': b'illegal value for "what" argument (%s): must be "changelog" or "changelog+rootmanifestlog"'}, b'status': b'error'}]
+
+Obtaining just the changelog works
+
+  $ sendhttpv2peer << EOF
+  > command rawstorefile
+  >     what changelog
+  > EOF
+  creating http peer for wire protocol version 2
+  sending rawstorefile command
+  s>     POST /api/exp-http-v2-0001/ro/rawstorefile HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     accept: application/mercurial-exp-framing-0005\r\n
+  s>     content-type: application/mercurial-exp-framing-0005\r\n
+  s>     content-length: 48\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     user-agent: Mercurial debugwireproto\r\n
+  s>     \r\n
+  s>     (\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa1DwhatIchangelogDnameLrawstorefile
+  s> makefile('rb', None)
+  s>     HTTP/1.1 200 OK\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Type: application/mercurial-exp-framing-0005\r\n
+  s>     Transfer-Encoding: chunked\r\n
+  s>     \r\n
+  s>     13\r\n
+  s>     \x0b\x00\x00\x01\x00\x02\x011
+  s>     \xa1FstatusBok
+  s>     \r\n
+  received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
+  s>     2e4\r\n
+  s>     \xdc\x02\x00\x01\x00\x02\x000
+  s>     \xa2Ifilecount\x01Itotalsize\x19\x02\xa2\xa2DnameM00changelog.iDsize\x19\x02\xa2_Y\x02\xa2\x00\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00M\x00\x00\x00N\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfd\x8dO6\xd3@\xdb\x7f\xa4Y\x10\x078k\xac\x0c\xc1\xb2#\xd0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00x\x9c%\xc5\xbb\r
+  s>     \x800\x0c\x05\xc0\xfeM\xc1\x0686!\xf18\xce\xc7\x92%D\x01\xd9_\x14\\s*f\xa7sME[\xa1,\xb3\xa9\xf8Q\x9d\x93\x0b[\xe6\xd4U\xbaQ\xc1\x9a\xef\x02m\x04CC\xc7\x88\x87\xf6\xf17\x81\xb8c\x85]\x1f+i\x15B\x00\x00\x00\x00\x00M\x00\x00\x00\x00\x00>\x00\x00\x00=\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\xff\xff\xff\xff\x04\x03\xe9K\x8c\xe4e\xa1\xf6\x99;\xb1\xe4`{\xbe\x17`V\xa7\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u356327300696aeacd7285a8755cea8447b237070\n
+  s>     test\n
+  s>     0 0\n
+  s>     a\n
+  s>     \n
+  s>     commit 2\x00\x00\x00\x00\x00\x8b\x00\x00\x00\x00\x00>\x00\x00\x00=\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x01\xff\xff\xff\xff\x0e\xb5\x06\x83\x84\x08E\xf3\xa7R\x8a}\xf3+l\xe2\x1c1J\x19\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u9ff26517c62e7379b7e07457817a1c9602aad8d6\n
+  s>     test\n
+  s>     0 0\n
+  s>     a\n
+  s>     \n
+  s>     commit 3\x00\x00\x00\x00\x00\xc9\x00\x00\x00\x00\x00I\x00\x00\x00H\x00\x00\x00\x03\x00\x00\x00\x03\x00\x00\x00\x00\xff\xff\xff\xff\x15_\xcd\xfdBs\xeb\xeb\\.T\xd1\xc8q\x1f\xfd\xc2\x993\xab\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u642dcaababdeb2f602127859e230e7859ca538b3\n
+  s>     test\n
+  s>     0 0\n
+  s>     a\n
+  s>     \n
+  s>     commit 4 (new head)\x00\x00\x00\x00\x01\x12\x00\x00\x00\x00\x00P\x00\x00\x00O\x00\x00\x00\x04\x00\x00\x00\x04\x00\x00\x00\x03\xff\xff\xff\xff\x9eE\xd8%Y&G)\xb4r\xc5\x167\xde\xdc\xf0\x99>\xb0\x8a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u19e0c69382fe33ca7026a1e2dfe59524b27a2b8f\n
+  s>     test\n
+  s>     0 0\n
+  s>     c\n
+  s>     c-moved\n
+  s>     \n
+  s>     commit 5 (moved c)\xff
+  s>     \r\n
+  received frame(size=732; request=1; stream=2; streamflags=; type=command-response; flags=)
+  s>     8\r\n
+  s>     \x00\x00\x00\x01\x00\x02\x002
+  s>     \r\n
+  s>     0\r\n
+  s>     \r\n
+  received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+  response: [{b'status': b'ok'}, {b'filecount': 1, b'totalsize': 674}, {b'name': b'00changelog.i', b'size': 674}, bytearray['\x00\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00M\x00\x00\x00N\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfd\x8dO6\xd3@\xdb\x7f\xa4Y\x10\x078k\xac\x0c\xc1\xb2#\xd0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00x\x9c%\xc5\xbb\r\x800\x0c\x05\xc0\xfeM\xc1\x0686!\xf18\xce\xc7\x92%D\x01\xd9_\x14\\s*f\xa7sME[\xa1,\xb3\xa9\xf8Q\x9d\x93\x0b[\xe6\xd4U\xbaQ\xc1\x9a\xef\x02m\x04CC\xc7\x88\x87\xf6\xf17\x81\xb8c\x85]\x1f+i\x15B\x00\x00\x00\x00\x00M\x00\x00\x00\x00\x00>\x00\x00\x00=\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\xff\xff\xff\xff\x04\x03\xe9K\x8c\xe4e\xa1\xf6\x99;\xb1\xe4`{\xbe\x17`V\xa7\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u356327300696aeacd7285a8755cea8447b237070\ntest\n0 0\na\n\ncommit 2\x00\x00\x00\x00\x00\x8b\x00\x00\x00\x00\x00>\x00\x00\x00=\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x01\xff\xff\xff\xff\x0e\xb5\x06\x83\x84\x08E\xf3\xa7R\x8a}\xf3+l\xe2\x1c1J\x19\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u9ff26517c62e7379b7e07457817a1c9602aad8d6\ntest\n0 0\na\n\ncommit 3\x00\x00\x00\x00\x00\xc9\x00\x00\x00\x00\x00I\x00\x00\x00H\x00\x00\x00\x03\x00\x00\x00\x03\x00\x00\x00\x00\xff\xff\xff\xff\x15_\xcd\xfdBs\xeb\xeb\\.T\xd1\xc8q\x1f\xfd\xc2\x993\xab\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u642dcaababdeb2f602127859e230e7859ca538b3\ntest\n0 0\na\n\ncommit 4 (new head)\x00\x00\x00\x00\x01\x12\x00\x00\x00\x00\x00P\x00\x00\x00O\x00\x00\x00\x04\x00\x00\x00\x04\x00\x00\x00\x03\xff\xff\xff\xff\x9eE\xd8%Y&G)\xb4r\xc5\x167\xde\xdc\xf0\x99>\xb0\x8a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u19e0c69382fe33ca7026a1e2dfe59524b27a2b8f\ntest\n0 0\nc\nc-moved\n\ncommit 5 (moved c)']]
+
+Obtaining changelog and the manifest works
+
+  $ sendhttpv2peer << EOF
+  > command rawstorefile
+  >     what changelog+rootmanifestlog
+  > EOF
+  creating http peer for wire protocol version 2
+  sending rawstorefile command
+  s>     POST /api/exp-http-v2-0001/ro/rawstorefile HTTP/1.1\r\n
+  s>     Accept-Encoding: identity\r\n
+  s>     accept: application/mercurial-exp-framing-0005\r\n
+  s>     content-type: application/mercurial-exp-framing-0005\r\n
+  s>     content-length: 65\r\n
+  s>     host: $LOCALIP:$HGPORT\r\n (glob)
+  s>     user-agent: Mercurial debugwireproto\r\n
+  s>     \r\n
+  s>     9\x00\x00\x01\x00\x01\x01\x11\xa2Dargs\xa1DwhatX\x19changelog+rootmanifestlogDnameLrawstorefile
+  s> makefile('rb', None)
+  s>     HTTP/1.1 200 OK\r\n
+  s>     Server: testing stub value\r\n
+  s>     Date: $HTTP_DATE$\r\n
+  s>     Content-Type: application/mercurial-exp-framing-0005\r\n
+  s>     Transfer-Encoding: chunked\r\n
+  s>     \r\n
+  s>     13\r\n
+  s>     \x0b\x00\x00\x01\x00\x02\x011
+  s>     \xa1FstatusBok
+  s>     \r\n
+  received frame(size=11; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=continuation)
+  s>     5be\r\n
+  s>     \xb6\x05\x00\x01\x00\x02\x000
+  s>     \xa2Ifilecount\x02Itotalsize\x19\x05\\\xa2DnameL00manifest.iDsize\x19\x02\xba_Y\x02\xba\x00\x03\x00\x01\x00\x00\x00\x00\x00\x00\x00\x98\x00\x00\x00\xe1\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\x93\xaao(\x17\x9bpS\xeb\x93\xf4\x8f!\xf3*R\x1c\x93\xca\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00x\x9c%\xcd1\x8e\x041\x08\x05Q\xc7}\x99\x01\xfc\xc1\xf6q\xc0\x804\xe9\xde?\xd8\x96&\x7f\xaa\xf2!\x81\n
+  s>     Z\x93O\xb4\xd3Z\x0eri\xc2\xe1iz\xbc\x9a\xd2!\xe9O\x8c\xcd\xa7DwNv-62\x93\xd3\xaf\x8a8$\xce!\\\x05a{\xee8\x0c@\'\xcc\xe9:\xbd.\x12\xebVhv\xd1*$n\x8b\x12?\xf9\xfd\xa3O\x0e\x9d[\xc8\xf2\x9e\xc5\xa5\xc2\n
+  s>     J\xdb\x98\x1e]\x96\xefjJ[B\xec\xe7k\xec\x15\xcau\xde\xb2Z\x91w\xc3H\xfd\xf2\xcd\xd5\xda\x9b\x80;9=\x9e\x7f\xb8\xcc:\xbf\x00\x00\x00\x00\x00\x98\x00\x00\x00\x00\x007\x00\x00\x00\xe1\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\xff\xff\xff\xff5c\'0\x06\x96\xae\xac\xd7(Z\x87U\xce\xa8D{#pp\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00+\x00\x00\x00+a\x009a38122997b3ac97be2a9aa2e556838341fdf2cc\n
+  s>     \x00\x00\x00\x00\x00\xcf\x00\x00\x00\x00\x007\x00\x00\x00\xe1\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x01\xff\xff\xff\xff\x9f\xf2e\x17\xc6.sy\xb7\xe0tW\x81z\x1c\x96\x02\xaa\xd8\xd6\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00+\x00\x00\x00+a\x00c2a205c8b2ade24af26062e53cd5bc3801d660da\n
+  s>     \x00\x00\x00\x00\x01\x06\x00\x00\x00\x00\x007\x00\x00\x00\xe1\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\xff\xff\xff\xffd-\xca\xab\xab\xde\xb2\xf6\x02\x12xY\xe20\xe7\x85\x9c\xa58\xb3\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00+\x00\x00\x00+a\x0036b617abfb829883cdd60767d165ea454dd81726\n
+  s>     \x00\x00\x00\x00\x01=\x00\x00\x00\x00\x00=\x00\x00\x00\xe7\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x03\xff\xff\xff\xff\x19\xe0\xc6\x93\x82\xfe3\xcap&\xa1\xe2\xdf\xe5\x95$\xb2z+\x8f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00V\x00\x00\x00\x81\x00\x00\x001c-moved\x00de51c302cf94dafe23574832ce72086e984acdde\n
+  s>     \xff\xa2DnameM00changelog.iDsize\x19\x02\xa2_Y\x02\xa2\x00\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00M\x00\x00\x00N\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfd\x8dO6\xd3@\xdb\x7f\xa4Y\x10\x078k\xac\x0c\xc1\xb2#\xd0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00x\x9c%\xc5\xbb\r
+  s>     \x800\x0c\x05\xc0\xfeM\xc1\x0686!\xf18\xce\xc7\x92%D\x01\xd9_\x14\\s*f\xa7sME[\xa1,\xb3\xa9\xf8Q\x9d\x93\x0b[\xe6\xd4U\xbaQ\xc1\x9a\xef\x02m\x04CC\xc7\x88\x87\xf6\xf17\x81\xb8c\x85]\x1f+i\x15B\x00\x00\x00\x00\x00M\x00\x00\x00\x00\x00>\x00\x00\x00=\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\xff\xff\xff\xff\x04\x03\xe9K\x8c\xe4e\xa1\xf6\x99;\xb1\xe4`{\xbe\x17`V\xa7\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u356327300696aeacd7285a8755cea8447b237070\n
+  s>     test\n
+  s>     0 0\n
+  s>     a\n
+  s>     \n
+  s>     commit 2\x00\x00\x00\x00\x00\x8b\x00\x00\x00\x00\x00>\x00\x00\x00=\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x01\xff\xff\xff\xff\x0e\xb5\x06\x83\x84\x08E\xf3\xa7R\x8a}\xf3+l\xe2\x1c1J\x19\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u9ff26517c62e7379b7e07457817a1c9602aad8d6\n
+  s>     test\n
+  s>     0 0\n
+  s>     a\n
+  s>     \n
+  s>     commit 3\x00\x00\x00\x00\x00\xc9\x00\x00\x00\x00\x00I\x00\x00\x00H\x00\x00\x00\x03\x00\x00\x00\x03\x00\x00\x00\x00\xff\xff\xff\xff\x15_\xcd\xfdBs\xeb\xeb\\.T\xd1\xc8q\x1f\xfd\xc2\x993\xab\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u642dcaababdeb2f602127859e230e7859ca538b3\n
+  s>     test\n
+  s>     0 0\n
+  s>     a\n
+  s>     \n
+  s>     commit 4 (new head)\x00\x00\x00\x00\x01\x12\x00\x00\x00\x00\x00P\x00\x00\x00O\x00\x00\x00\x04\x00\x00\x00\x04\x00\x00\x00\x03\xff\xff\xff\xff\x9eE\xd8%Y&G)\xb4r\xc5\x167\xde\xdc\xf0\x99>\xb0\x8a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u19e0c69382fe33ca7026a1e2dfe59524b27a2b8f\n
+  s>     test\n
+  s>     0 0\n
+  s>     c\n
+  s>     c-moved\n
+  s>     \n
+  s>     commit 5 (moved c)\xff
+  s>     \r\n
+  received frame(size=1462; request=1; stream=2; streamflags=; type=command-response; flags=)
+  s>     8\r\n
+  s>     \x00\x00\x00\x01\x00\x02\x002
+  s>     \r\n
+  s>     0\r\n
+  s>     \r\n
+  received frame(size=0; request=1; stream=2; streamflags=; type=command-response; flags=eos)
+  response: [{b'status': b'ok'}, {b'filecount': 2, b'totalsize': 1372}, {b'name': b'00manifest.i', b'size': 698}, bytearray['\x00\x03\x00\x01\x00\x00\x00\x00\x00\x00\x00\x98\x00\x00\x00\xe1\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\x93\xaao(\x17\x9bpS\xeb\x93\xf4\x8f!\xf3*R\x1c\x93\xca\x07\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00x\x9c%\xcd1\x8e\x041\x08\x05Q\xc7}\x99\x01\xfc\xc1\xf6q\xc0\x804\xe9\xde?\xd8\x96&\x7f\xaa\xf2!\x81\nZ\x93O\xb4\xd3Z\x0eri\xc2\xe1iz\xbc\x9a\xd2!\xe9O\x8c\xcd\xa7DwNv-62\x93\xd3\xaf\x8a8$\xce!\\\x05a{\xee8\x0c@\'\xcc\xe9:\xbd.\x12\xebVhv\xd1*$n\x8b\x12?\xf9\xfd\xa3O\x0e\x9d[\xc8\xf2\x9e\xc5\xa5\xc2\nJ\xdb\x98\x1e]\x96\xefjJ[B\xec\xe7k\xec\x15\xcau\xde\xb2Z\x91w\xc3H\xfd\xf2\xcd\xd5\xda\x9b\x80;9=\x9e\x7f\xb8\xcc:\xbf\x00\x00\x00\x00\x00\x98\x00\x00\x00\x00\x007\x00\x00\x00\xe1\x00\x00\x00\x00\x00\x00\x00\x01\x00\x00\x00\x00\xff\xff\xff\xff5c\'0\x06\x96\xae\xac\xd7(Z\x87U\xce\xa8D{#pp\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00+\x00\x00\x00+a\x009a38122997b3ac97be2a9aa2e556838341fdf2cc\n\x00\x00\x00\x00\x00\xcf\x00\x00\x00\x00\x007\x00\x00\x00\xe1\x00\x00\x00\x01\x00\x00\x00\x02\x00\x00\x00\x01\xff\xff\xff\xff\x9f\xf2e\x17\xc6.sy\xb7\xe0tW\x81z\x1c\x96\x02\xaa\xd8\xd6\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00+\x00\x00\x00+a\x00c2a205c8b2ade24af26062e53cd5bc3801d660da\n\x00\x00\x00\x00\x01\x06\x00\x00\x00\x00\x007\x00\x00\x00\xe1\x00\x00\x00\x00\x00\x00\x00\x03\x00\x00\x00\x00\xff\xff\xff\xffd-\xca\xab\xab\xde\xb2\xf6\x02\x12xY\xe20\xe7\x85\x9c\xa58\xb3\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00+\x00\x00\x00+a\x0036b617abfb829883cdd60767d165ea454dd81726\n\x00\x00\x00\x00\x01=\x00\x00\x00\x00\x00=\x00\x00\x00\xe7\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x03\xff\xff\xff\xff\x19\xe0\xc6\x93\x82\xfe3\xcap&\xa1\xe2\xdf\xe5\x95$\xb2z+\x8f\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00V\x00\x00\x00\x81\x00\x00\x001c-moved\x00de51c302cf94dafe23574832ce72086e984acdde\n'], {b'name': b'00changelog.i', b'size': 674}, bytearray['\x00\x01\x00\x01\x00\x00\x00\x00\x00\x00\x00M\x00\x00\x00N\x00\x00\x00\x00\x00\x00\x00\x00\xff\xff\xff\xff\xff\xff\xff\xff\xfd\x8dO6\xd3@\xdb\x7f\xa4Y\x10\x078k\xac\x0c\xc1\xb2#\xd0\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00x\x9c%\xc5\xbb\r\x800\x0c\x05\xc0\xfeM\xc1\x0686!\xf18\xce\xc7\x92%D\x01\xd9_\x14\\s*f\xa7sME[\xa1,\xb3\xa9\xf8Q\x9d\x93\x0b[\xe6\xd4U\xbaQ\xc1\x9a\xef\x02m\x04CC\xc7\x88\x87\xf6\xf17\x81\xb8c\x85]\x1f+i\x15B\x00\x00\x00\x00\x00M\x00\x00\x00\x00\x00>\x00\x00\x00=\x00\x00\x00\x01\x00\x00\x00\x01\x00\x00\x00\x00\xff\xff\xff\xff\x04\x03\xe9K\x8c\xe4e\xa1\xf6\x99;\xb1\xe4`{\xbe\x17`V\xa7\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u356327300696aeacd7285a8755cea8447b237070\ntest\n0 0\na\n\ncommit 2\x00\x00\x00\x00\x00\x8b\x00\x00\x00\x00\x00>\x00\x00\x00=\x00\x00\x00\x02\x00\x00\x00\x02\x00\x00\x00\x01\xff\xff\xff\xff\x0e\xb5\x06\x83\x84\x08E\xf3\xa7R\x8a}\xf3+l\xe2\x1c1J\x19\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u9ff26517c62e7379b7e07457817a1c9602aad8d6\ntest\n0 0\na\n\ncommit 3\x00\x00\x00\x00\x00\xc9\x00\x00\x00\x00\x00I\x00\x00\x00H\x00\x00\x00\x03\x00\x00\x00\x03\x00\x00\x00\x00\xff\xff\xff\xff\x15_\xcd\xfdBs\xeb\xeb\\.T\xd1\xc8q\x1f\xfd\xc2\x993\xab\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u642dcaababdeb2f602127859e230e7859ca538b3\ntest\n0 0\na\n\ncommit 4 (new head)\x00\x00\x00\x00\x01\x12\x00\x00\x00\x00\x00P\x00\x00\x00O\x00\x00\x00\x04\x00\x00\x00\x04\x00\x00\x00\x03\xff\xff\xff\xff\x9eE\xd8%Y&G)\xb4r\xc5\x167\xde\xdc\xf0\x99>\xb0\x8a\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00u19e0c69382fe33ca7026a1e2dfe59524b27a2b8f\ntest\n0 0\nc\nc-moved\n\ncommit 5 (moved c)']]
+
+  $ cat error.log
diff --git a/tests/test-wireproto-command-capabilities.t b/tests/test-wireproto-command-capabilities.t
--- a/tests/test-wireproto-command-capabilities.t
+++ b/tests/test-wireproto-command-capabilities.t
@@ -202,8 +202,8 @@ 
   s>     Content-Type: application/mercurial-cbor\r\n
   s>     Content-Length: *\r\n (glob)
   s>     \r\n
-  s>     \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xa7Eheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
-  cbor> {b'apibase': b'api/', b'apis': {b'exp-http-v2-0001': {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}}, b'v1capabilities': b'batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash'}
+  s>     \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xa8Eheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
+  cbor> {b'apibase': b'api/', b'apis': {b'exp-http-v2-0001': {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}, b'rawstorefile': {b'args': {b'what': b'changelog'}, b'permissions': [b'pull']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}}, b'v1capabilities': b'batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash'}
 
 capabilities command returns expected info
 
@@ -227,7 +227,7 @@ 
   s>     Content-Type: application/mercurial-cbor\r\n
   s>     Content-Length: *\r\n (glob)
   s>     \r\n
-  s>     \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xa7Eheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
+  s>     \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xa8Eheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
   sending capabilities command
   s>     POST /api/exp-http-v2-0001/ro/capabilities HTTP/1.1\r\n
   s>     Accept-Encoding: identity\r\n
@@ -245,13 +245,13 @@ 
   s>     Content-Type: application/mercurial-exp-framing-0005\r\n
   s>     Transfer-Encoding: chunked\r\n
   s>     \r\n
-  s>     1d7\r\n
-  s>     \xcf\x01\x00\x01\x00\x02\x012
-  s>     \xa1FstatusBok\xa4Hcommands\xa7Eheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005
+  s>     20c\r\n
+  s>     \x04\x02\x00\x01\x00\x02\x012
+  s>     \xa1FstatusBok\xa4Hcommands\xa8Eheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005
   s>     \r\n
-  received frame(size=463; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=eos)
+  received frame(size=516; request=1; stream=2; streamflags=stream-begin; type=command-response; flags=eos)
   s>     0\r\n
   s>     \r\n
-  response: [{b'status': b'ok'}, {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}]
+  response: [{b'status': b'ok'}, {b'commands': {b'branchmap': {b'args': {}, b'permissions': [b'pull']}, b'capabilities': {b'args': {}, b'permissions': [b'pull']}, b'heads': {b'args': {b'publiconly': False}, b'permissions': [b'pull']}, b'known': {b'args': {b'nodes': [b'deadbeef']}, b'permissions': [b'pull']}, b'listkeys': {b'args': {b'namespace': b'ns'}, b'permissions': [b'pull']}, b'lookup': {b'args': {b'key': b'foo'}, b'permissions': [b'pull']}, b'pushkey': {b'args': {b'key': b'key', b'namespace': b'ns', b'new': b'new', b'old': b'old'}, b'permissions': [b'push']}, b'rawstorefile': {b'args': {b'what': b'changelog'}, b'permissions': [b'pull']}}, b'compression': [{b'name': b'zlib'}], b'framingmediatypes': [b'application/mercurial-exp-framing-0005'], b'rawrepoformats': [b'generaldelta', b'revlogv1']}]
 
   $ cat error.log
diff --git a/tests/test-http-protocol.t b/tests/test-http-protocol.t
--- a/tests/test-http-protocol.t
+++ b/tests/test-http-protocol.t
@@ -305,7 +305,7 @@ 
   s>     Content-Type: application/mercurial-cbor\r\n
   s>     Content-Length: *\r\n (glob)
   s>     \r\n
-  s>     \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xa7Eheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
+  s>     \xa3Dapis\xa1Pexp-http-v2-0001\xa4Hcommands\xa8Eheads\xa2Dargs\xa1Jpubliconly\xf4Kpermissions\x81DpullEknown\xa2Dargs\xa1Enodes\x81HdeadbeefKpermissions\x81DpullFlookup\xa2Dargs\xa1CkeyCfooKpermissions\x81DpullGpushkey\xa2Dargs\xa4CkeyCkeyCnewCnewColdColdInamespaceBnsKpermissions\x81DpushHlistkeys\xa2Dargs\xa1InamespaceBnsKpermissions\x81DpullIbranchmap\xa2Dargs\xa0Kpermissions\x81DpullLcapabilities\xa2Dargs\xa0Kpermissions\x81DpullLrawstorefile\xa2Dargs\xa1DwhatIchangelogKpermissions\x81DpullKcompression\x81\xa1DnameDzlibNrawrepoformats\x82LgeneraldeltaHrevlogv1Qframingmediatypes\x81X&application/mercurial-exp-framing-0005GapibaseDapi/Nv1capabilitiesY\x01\xc5batch branchmap $USUAL_BUNDLE2_CAPS_SERVER$ changegroupsubset compression=$BUNDLE2_COMPRESSIONS$ getbundle httpheader=1024 httpmediatype=0.1rx,0.1tx,0.2tx known lookup pushkey streamreqs=generaldelta,revlogv1 unbundle=HG10GZ,HG10BZ,HG10UN unbundlehash
   sending heads command
   s>     POST /api/exp-http-v2-0001/ro/heads HTTP/1.1\r\n
   s>     Accept-Encoding: identity\r\n
diff --git a/mercurial/wireprotov2server.py b/mercurial/wireprotov2server.py
--- a/mercurial/wireprotov2server.py
+++ b/mercurial/wireprotov2server.py
@@ -22,6 +22,7 @@ 
     wireprototypes,
 )
 from .utils import (
+    cborutil,
     interfaceutil,
 )
 
@@ -532,3 +533,61 @@ 
                      encoding.tolocal(new))
 
     return wireprototypes.cborresponse(r)
+
+@wireprotocommand('rawstorefile',
+                  args={
+                      'what': b'changelog',
+                  },
+                  permission='pull')
+def rawstorefile(repo, proto, what=None):
+    if what not in ('changelog', 'changelog+rootmanifestlog'):
+        return wireprototypes.v2errorresponse(
+            'illegal value for "what" argument (%s): must be "changelog" '
+            'or "changelog+rootmanifestlog"', (what,))
+
+    with repo.lock():
+        topfiles = list(repo.store.topfiles())
+
+    sendfiles = []
+    totalsize = 0
+
+    for name, encodedname, size in topfiles:
+        if what.startswith(b'changelog') and name.startswith(b'00changelog'):
+            pass
+        elif (what == b'changelog+rootmanifestlog' and
+              name.startswith(b'00manifest')):
+            pass
+        else:
+            continue
+
+        sendfiles.append((name, encodedname, size))
+        totalsize += size
+
+    def senddata():
+        chunks = cborutil.streamencodemap({
+            b'filecount': len(sendfiles),
+            b'totalsize': totalsize,
+        })
+
+        for chunk in chunks:
+            yield chunk
+
+        for name, encodedname, size in sendfiles:
+            chunks = cborutil.streamencodemap({
+                b'name': encodedname,
+                b'size': size,
+            })
+            for chunk in chunks:
+                yield chunk
+
+            with repo.svfs(name, 'rb', auditpath=False) as fh:
+                if size <= 65536:
+                    chunks = [fh.read(size)]
+                else:
+                    chunks = util.filechunkiter(fh, limit=size)
+
+                for chunk in cborutil.streamencodebytestringfromiter(chunks):
+                    yield chunk
+
+    # We assume we're serving revlogs and that they don't compress well.
+    return wireprototypes.v2streamingresponse(senddata(), compressible=False)
diff --git a/mercurial/help/internals/wireprotocol.txt b/mercurial/help/internals/wireprotocol.txt
--- a/mercurial/help/internals/wireprotocol.txt
+++ b/mercurial/help/internals/wireprotocol.txt
@@ -1911,3 +1911,40 @@ 
 
 TODO consider using binary to represent nodes is certain pushkey namespaces.
 TODO better define response type and meaning.
+
+rawstorefile
+------------
+
+Obtain raw files from the server.
+
+This command is used to quickly obtain files in the repository store
+from the remote with minimal additional processing.
+
+The command receives the following arguments:
+
+what
+   (bytestring) Type of query to perform. Can be ``changelog`` or
+   ``changelog+rootmanifestlog``.
+
+TODO need a mechanism to consider treemanifests
+
+The response begins with a CBOR map containing the following keys:
+
+filecount
+   (unsigned integer) Number of files being included in the payload.
+totalsize
+   (unsigned integer) Total size of all transferred file data, in bytes.
+
+Following this map are N file segments. Each file segment consists of
+a CBOR map followed by a CBOR bytestring, possibly of indefinite length.
+
+Each CBOR map contains the following bytestring keys:
+
+name
+   (bytestring) Path of file being transferred. Path is the raw store
+   path and can be any sequence of bytes that can be tracked in a
+   Mercurial manifest.
+size
+   (unsigned integer) Size of file data. This will be the final written
+   file size. The total size of the data that follows the CBOR map
+   will be greater due to encoding overhead of CBOR.