Patchwork [3,of,3] treemanifests: include dirlogs in streaming cloning

login
register
mail settings
Submitter via Mercurial-devel
Date Aug. 26, 2016, 5:47 p.m.
Message ID <f1618da9e37d9cd7d0a3.1472233654@martinvonz.mtv.corp.google.com>
Download mbox | patch
Permalink /patch/16458/
State Accepted
Headers show

Comments

via Mercurial-devel - Aug. 26, 2016, 5:47 p.m.
# HG changeset patch
# User Martin von Zweigbergk <martinvonz@google.com>
# Date 1472232129 25200
#      Fri Aug 26 10:22:09 2016 -0700
# Node ID f1618da9e37d9cd7d0a3bb3ae955c4b3f43f222e
# Parent  35fde8eff1569689667c21973fb7193b3ad049b8
treemanifests: include dirlogs in streaming cloning

We need to include the dirlogs (meta/**/00manifest.{i,d}) in streaming
clones for treemanifests to work. Also add a test for it.
Durham Goode - Aug. 29, 2016, 6:10 p.m.
Pushed the series. Thanks

On 8/26/16, 10:47 AM, "Martin von Zweigbergk" <martinvonz@google.com> wrote:

># HG changeset patch

># User Martin von Zweigbergk <martinvonz@google.com>

># Date 1472232129 25200

>#      Fri Aug 26 10:22:09 2016 -0700

># Node ID f1618da9e37d9cd7d0a3bb3ae955c4b3f43f222e

># Parent  35fde8eff1569689667c21973fb7193b3ad049b8

>treemanifests: include dirlogs in streaming cloning

>

>We need to include the dirlogs (meta/**/00manifest.{i,d}) in streaming

>clones for treemanifests to work. Also add a test for it.

>

>diff -r 35fde8eff156 -r f1618da9e37d remotefilelog/remotefilelogserver.py

>--- a/remotefilelog/remotefilelogserver.py	Fri Aug 26 10:20:10 2016 -0700

>+++ b/remotefilelog/remotefilelogserver.py	Fri Aug 26 10:22:09 2016 -0700

>@@ -125,6 +125,12 @@

>                         if kind == stat.S_IFDIR:

>                             visit.append(fp)

> 

>+            if 'treemanifest' in repo.requirements:

>+                for (u, e, s) in repo.store.datafiles():

>+                    if (u.startswith('meta/') and

>+                        (u.endswith('.i') or u.endswith('.d'))):

>+                        yield (u, e, s)

>+

>             # Return .d and .i files that do not match the shallow pattern

>             match = state.match

>             if match and not match.always():

>diff -r 35fde8eff156 -r f1618da9e37d tests/library.sh

>--- a/tests/library.sh	Fri Aug 26 10:20:10 2016 -0700

>+++ b/tests/library.sh	Fri Aug 26 10:22:09 2016 -0700

>@@ -14,6 +14,8 @@

> ssh=python "$TESTDIR/dummyssh"

> [server]

> preferuncompressed=True

>+[experimental]

>+changegroup3=True

> EOF

> export HGRCPATH=$PWD/.hgrc

> 

>diff -r 35fde8eff156 -r f1618da9e37d tests/test-clone-tree.t

>--- /dev/null	Thu Jan 01 00:00:00 1970 +0000

>+++ b/tests/test-clone-tree.t	Fri Aug 26 10:22:09 2016 -0700

>@@ -0,0 +1,113 @@

>+  $ . "$TESTDIR/library.sh"

>+

>+  $ hginit master

>+  $ cd master

>+  $ echo treemanifest >> .hg/requires

>+  $ cat >> .hg/hgrc <<EOF

>+  > [remotefilelog]

>+  > server=True

>+  > EOF

>+# uppercase directory name to test encoding

>+  $ mkdir -p A/B

>+  $ echo x > A/B/x

>+  $ hg commit -qAm x

>+

>+  $ cd ..

>+

>+# shallow clone from full

>+

>+  $ hgcloneshallow ssh://user@dummy/master shallow --noupdate

>+  streaming all changes

>+  4 files to transfer, 449 bytes of data

>+  transferred 449 bytes in * seconds (*/sec) (glob)

>+  searching for changes

>+  no changes found

>+  $ cd shallow

>+  $ cat .hg/requires

>+  dotencode

>+  fncache

>+  generaldelta

>+  remotefilelog

>+  revlogv1

>+  store

>+  treemanifest

>+  $ find .hg/store/meta

>+  .hg/store/meta

>+  .hg/store/meta/_a

>+  .hg/store/meta/_a/_b

>+  .hg/store/meta/_a/_b/00manifest.i

>+  .hg/store/meta/_a/00manifest.i

>+

>+  $ hg update

>+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

>+  1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)

>+

>+  $ cat A/B/x

>+  x

>+

>+  $ ls .hg/store/data

>+  $ echo foo > A/B/F

>+  $ hg add A/B/F

>+  $ hg ci -m 'local content'

>+  $ ls .hg/store/data

>+  ca31988f085bfb945cb8115b78fabdee40f741aa

>+

>+  $ cd ..

>+

>+# shallow clone from shallow

>+

>+  $ hgcloneshallow ssh://user@dummy/shallow shallow2  --noupdate

>+  streaming all changes

>+  5 files to transfer, 1008 bytes of data

>+  transferred 1008 bytes in * seconds (*/sec) (glob)

>+  searching for changes

>+  no changes found

>+  $ cd shallow2

>+  $ cat .hg/requires

>+  dotencode

>+  fncache

>+  generaldelta

>+  remotefilelog

>+  revlogv1

>+  store

>+  treemanifest

>+  $ ls .hg/store/data

>+  ca31988f085bfb945cb8115b78fabdee40f741aa

>+

>+  $ hg update

>+  2 files updated, 0 files merged, 0 files removed, 0 files unresolved

>+

>+  $ cat A/B/x

>+  x

>+

>+  $ cd ..

>+

>+# full clone from shallow

>+

>+  $ hg clone --noupdate ssh://user@dummy/shallow full

>+  streaming all changes

>+  remote: abort: Cannot clone from a shallow repo to a full repo.

>+  abort: unexpected response from remote server: empty string

>+  [255]

>+

>+# getbundle full clone

>+

>+  $ printf '[server]\npreferuncompressed=False\n' >> master/.hg/hgrc

>+  $ hgcloneshallow ssh://user@dummy/master shallow3

>+  requesting all changes

>+  adding changesets

>+  adding manifests

>+  adding file changes

>+  added 1 changesets with 0 changes to 0 files

>+  updating to branch default

>+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved

>+

>+  $ ls shallow3/.hg/store/data

>+  $ cat shallow3/.hg/requires

>+  dotencode

>+  fncache

>+  generaldelta

>+  remotefilelog

>+  revlogv1

>+  store

>+  treemanifest

>diff -r 35fde8eff156 -r f1618da9e37d tests/test-http.t

>--- a/tests/test-http.t	Fri Aug 26 10:20:10 2016 -0700

>+++ b/tests/test-http.t	Fri Aug 26 10:22:09 2016 -0700

>@@ -37,11 +37,11 @@

>   $ get-with-headers.py localhost:$HGPORT '?cmd=capabilities'

>   200 Script output follows

>   

>-  lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch stream-preferred streamreqs=generaldelta,revlogv1 bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 getflogheads getfile (no-eol)

>+  lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch stream-preferred streamreqs=generaldelta,revlogv1 bundle2=HG20%0Achangegroup%3D01%2C02%2C03%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 getflogheads getfile (no-eol)

>   $ get-with-headers.py localhost:$HGPORT '?cmd=hello'

>   200 Script output follows

>   

>-  capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch stream-preferred streamreqs=generaldelta,revlogv1 bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 getflogheads getfile

>+  capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch stream-preferred streamreqs=generaldelta,revlogv1 bundle2=HG20%0Achangegroup%3D01%2C02%2C03%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 getflogheads getfile

> 

>   $ get-with-headers.py localhost:$HGPORT '?cmd=this-command-does-not-exist' | head -n 1

>   400 no such method: this-command-does-not-exist

Patch

diff -r 35fde8eff156 -r f1618da9e37d remotefilelog/remotefilelogserver.py
--- a/remotefilelog/remotefilelogserver.py	Fri Aug 26 10:20:10 2016 -0700
+++ b/remotefilelog/remotefilelogserver.py	Fri Aug 26 10:22:09 2016 -0700
@@ -125,6 +125,12 @@ 
                         if kind == stat.S_IFDIR:
                             visit.append(fp)
 
+            if 'treemanifest' in repo.requirements:
+                for (u, e, s) in repo.store.datafiles():
+                    if (u.startswith('meta/') and
+                        (u.endswith('.i') or u.endswith('.d'))):
+                        yield (u, e, s)
+
             # Return .d and .i files that do not match the shallow pattern
             match = state.match
             if match and not match.always():
diff -r 35fde8eff156 -r f1618da9e37d tests/library.sh
--- a/tests/library.sh	Fri Aug 26 10:20:10 2016 -0700
+++ b/tests/library.sh	Fri Aug 26 10:22:09 2016 -0700
@@ -14,6 +14,8 @@ 
 ssh=python "$TESTDIR/dummyssh"
 [server]
 preferuncompressed=True
+[experimental]
+changegroup3=True
 EOF
 export HGRCPATH=$PWD/.hgrc
 
diff -r 35fde8eff156 -r f1618da9e37d tests/test-clone-tree.t
--- /dev/null	Thu Jan 01 00:00:00 1970 +0000
+++ b/tests/test-clone-tree.t	Fri Aug 26 10:22:09 2016 -0700
@@ -0,0 +1,113 @@ 
+  $ . "$TESTDIR/library.sh"
+
+  $ hginit master
+  $ cd master
+  $ echo treemanifest >> .hg/requires
+  $ cat >> .hg/hgrc <<EOF
+  > [remotefilelog]
+  > server=True
+  > EOF
+# uppercase directory name to test encoding
+  $ mkdir -p A/B
+  $ echo x > A/B/x
+  $ hg commit -qAm x
+
+  $ cd ..
+
+# shallow clone from full
+
+  $ hgcloneshallow ssh://user@dummy/master shallow --noupdate
+  streaming all changes
+  4 files to transfer, 449 bytes of data
+  transferred 449 bytes in * seconds (*/sec) (glob)
+  searching for changes
+  no changes found
+  $ cd shallow
+  $ cat .hg/requires
+  dotencode
+  fncache
+  generaldelta
+  remotefilelog
+  revlogv1
+  store
+  treemanifest
+  $ find .hg/store/meta
+  .hg/store/meta
+  .hg/store/meta/_a
+  .hg/store/meta/_a/_b
+  .hg/store/meta/_a/_b/00manifest.i
+  .hg/store/meta/_a/00manifest.i
+
+  $ hg update
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  1 files fetched over 1 fetches - (1 misses, 0.00% hit ratio) over *s (glob)
+
+  $ cat A/B/x
+  x
+
+  $ ls .hg/store/data
+  $ echo foo > A/B/F
+  $ hg add A/B/F
+  $ hg ci -m 'local content'
+  $ ls .hg/store/data
+  ca31988f085bfb945cb8115b78fabdee40f741aa
+
+  $ cd ..
+
+# shallow clone from shallow
+
+  $ hgcloneshallow ssh://user@dummy/shallow shallow2  --noupdate
+  streaming all changes
+  5 files to transfer, 1008 bytes of data
+  transferred 1008 bytes in * seconds (*/sec) (glob)
+  searching for changes
+  no changes found
+  $ cd shallow2
+  $ cat .hg/requires
+  dotencode
+  fncache
+  generaldelta
+  remotefilelog
+  revlogv1
+  store
+  treemanifest
+  $ ls .hg/store/data
+  ca31988f085bfb945cb8115b78fabdee40f741aa
+
+  $ hg update
+  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
+
+  $ cat A/B/x
+  x
+
+  $ cd ..
+
+# full clone from shallow
+
+  $ hg clone --noupdate ssh://user@dummy/shallow full
+  streaming all changes
+  remote: abort: Cannot clone from a shallow repo to a full repo.
+  abort: unexpected response from remote server: empty string
+  [255]
+
+# getbundle full clone
+
+  $ printf '[server]\npreferuncompressed=False\n' >> master/.hg/hgrc
+  $ hgcloneshallow ssh://user@dummy/master shallow3
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 0 changes to 0 files
+  updating to branch default
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+
+  $ ls shallow3/.hg/store/data
+  $ cat shallow3/.hg/requires
+  dotencode
+  fncache
+  generaldelta
+  remotefilelog
+  revlogv1
+  store
+  treemanifest
diff -r 35fde8eff156 -r f1618da9e37d tests/test-http.t
--- a/tests/test-http.t	Fri Aug 26 10:20:10 2016 -0700
+++ b/tests/test-http.t	Fri Aug 26 10:22:09 2016 -0700
@@ -37,11 +37,11 @@ 
   $ get-with-headers.py localhost:$HGPORT '?cmd=capabilities'
   200 Script output follows
   
-  lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch stream-preferred streamreqs=generaldelta,revlogv1 bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 getflogheads getfile (no-eol)
+  lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch stream-preferred streamreqs=generaldelta,revlogv1 bundle2=HG20%0Achangegroup%3D01%2C02%2C03%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 getflogheads getfile (no-eol)
   $ get-with-headers.py localhost:$HGPORT '?cmd=hello'
   200 Script output follows
   
-  capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch stream-preferred streamreqs=generaldelta,revlogv1 bundle2=HG20%0Achangegroup%3D01%2C02%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 getflogheads getfile
+  capabilities: lookup changegroupsubset branchmap pushkey known getbundle unbundlehash batch stream-preferred streamreqs=generaldelta,revlogv1 bundle2=HG20%0Achangegroup%3D01%2C02%2C03%0Adigests%3Dmd5%2Csha1%2Csha512%0Aerror%3Dabort%2Cunsupportedcontent%2Cpushraced%2Cpushkey%0Ahgtagsfnodes%0Alistkeys%0Apushkey%0Aremote-changegroup%3Dhttp%2Chttps unbundle=HG10GZ,HG10BZ,HG10UN httpheader=1024 getflogheads getfile
 
   $ get-with-headers.py localhost:$HGPORT '?cmd=this-command-does-not-exist' | head -n 1
   400 no such method: this-command-does-not-exist