Patchwork [V2] remotenames: selectivepull, pull only bookmarks that are in remotenames

login
register
mail settings
Submitter Stanislau Hlebik
Date Aug. 18, 2016, 9:51 a.m.
Message ID <069b8a5f40ec450aebcc.1471513910@dev1918.lla1.facebook.com>
Download mbox | patch
Permalink /patch/16350/
State Superseded
Headers show

Comments

Stanislau Hlebik - Aug. 18, 2016, 9:51 a.m.
# HG changeset patch
# User Stanislau Hlebik <stash@fb.com>
# Date 1471513820 25200
#      Thu Aug 18 02:50:20 2016 -0700
# Node ID 069b8a5f40ec450aebccd2658ec49a1c694687a2
# Parent  7a6c5ff76f225c8ebe9baef9d5ef753da915aa8b
remotenames: selectivepull, pull only bookmarks that are in remotenames

Initial diff for the `Selective Pull` project (for details see https://www.mercurial-scm.org/wiki/SelectivePullPlan).
A little bit of context. There are certain scalability issues if hg server has many bookmarks.
And usually user is interested in only a small number of bookmarks.  This project will solve
these scalability issues by downloading only bookmarks in which user is "interested".
By default user is interested in `remotenames.selectivepulldefault` bookmark.
If user does `hg pull -B remotebook`, then user is also interested in `remotebook`
(in later diffs user will be able to show interest in bookmarks by running `hg up BOOK`).

There are behavior changes with remotenames.selectivepull turned on:
1 When running `hg pull` if `-B` option is not specified then remotenames file is parsed and only bookmarks
  for this remote are downloaded (or `master` if there are no bookmarks for this remote in remotenames file)
2 Only "interesting" bookmarks are saved in remotenames file.

There were a few questions/suggestions from Sean Farley.
1 Sean suggested to add `hg pull --all` flag to override selectivepull behavior.
  I'll add it in the next diffs.
2 Pass a refspec as a parameter. I haven't thought about it, but we can add it
  later if it's necessary.
3 Use `remotenames.alias.default` instead of `remotenames.selectivepull`. But
  for me it seems that these two flags have different purpose. So I'm not sure
  why we should mix them.

Test Plan:
Run remotenames tests

Patch

diff --git a/remotenames.py b/remotenames.py
--- a/remotenames.py
+++ b/remotenames.py
@@ -74,7 +74,38 @@ 
         setattr(pushop, flag, kwargs.get(flag))
 
 def expull(orig, repo, remote, *args, **kwargs):
-    bookmarks = remote.listkeys('bookmarks')
+    isselectivepull = repo.ui.configbool('remotenames', 'selectivepull', False)
+    remotebookmarks = remote.listkeys('bookmarks')
+    if isselectivepull:
+        path = activepath(repo.ui, remote)
+        bookmarks = {}
+        for bookmark in readbookmarknames(repo, path):
+            if bookmark in remotebookmarks:
+                bookmarks[bookmark] = remotebookmarks[bookmark]
+        if not bookmarks:
+            default_book = repo.ui.config('remotenames', 'selectivepulldefault')
+            if not default_book:
+                raise error.Abort(
+                    _('no default bookmark specified for selectivepull'))
+            if default_book in remotebookmarks:
+                bookmarks = {default_book: remotebookmarks[default_book]}
+            else:
+                raise error.Abort(
+                    _('default bookmark %s is not found on remote') %
+                    default_book)
+
+        if kwargs.get('bookmarks'):
+            for bookmark in kwargs['bookmarks']:
+                bookmarks[bookmark] = remotebookmarks[bookmark]
+        else:
+            heads = kwargs.get('heads') or []
+            for bookmark in bookmarks:
+                heads.append(remote.lookup(remotebookmarks[bookmark]))
+            kwargs['bookmarks'] = bookmarks
+            kwargs['heads'] = heads
+    else:
+        bookmarks = remotebookmarks
+
     res = orig(repo, remote, *args, **kwargs)
     pullremotenames(repo, remote, bookmarks)
     return res
@@ -1239,6 +1270,11 @@ 
     else:
         return repo.vfs
 
+def readbookmarknames(repo, remote):
+    for node, nametype, remotename, rname in readremotenames(repo):
+        if nametype == 'bookmarks' and remotename == remote:
+            yield rname
+
 def readremotenames(repo):
     vfs = shareawarevfs(repo)
     # exit early if there is nothing to do
diff --git a/tests/test-selective-pull.t b/tests/test-selective-pull.t
new file mode 100644
--- /dev/null
+++ b/tests/test-selective-pull.t
@@ -0,0 +1,158 @@ 
+Set up repos
+
+  $ cat >> $HGRCPATH << EOF
+  > [ui]
+  > ssh=python "$TESTDIR/dummyssh"
+  > [extensions]
+  > remotenames=`dirname $TESTDIR`/remotenames.py
+  > EOF
+  $ hg init remoterepo
+  $ hg clone -q ssh://user@dummy/remoterepo localrepo
+
+Pull master bookmark
+
+  $ cd remoterepo
+  $ echo a > a
+  $ hg add a
+  $ hg commit -m 'First'
+  $ hg book master
+  $ cd ../localrepo
+  $ hg pull
+  pulling from ssh://user@dummy/remoterepo
+  requesting all changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  (run 'hg update' to get a working copy)
+  $ hg bookmarks --remote
+     default/master            0:1449e7934ec1
+
+Set up selective pull
+  $ cat >> $HGRCPATH << EOF
+  > [remotenames]
+  > selectivepull=True
+  > selectivepulldefault=master
+  > EOF
+
+Create another bookmark on the remote repo
+  $ cd ../remoterepo
+  $ hg up null
+  0 files updated, 0 files merged, 1 files removed, 0 files unresolved
+  (leaving bookmark master)
+  $ hg book secondbook
+  $ echo b >> a
+  $ hg add a
+  $ hg commit -m 'Commit secondbook points to'
+  created new head
+
+Do not pull new boookmark from local repo
+  $ cd ../localrepo
+  $ hg pull
+  pulling from ssh://user@dummy/remoterepo
+  no changes found
+  $ hg bookmarks --remote
+     default/master            0:1449e7934ec1
+
+Do not pull new bookmark even if it on the same commit as old bookmark
+  $ cd ../remoterepo
+  $ hg up -q master
+  $ hg book thirdbook
+  $ cd ../localrepo
+  $ hg pull
+  pulling from ssh://user@dummy/remoterepo
+  no changes found
+  $ hg bookmarks --remote
+     default/master            0:1449e7934ec1
+
+Move master bookmark
+  $ cd ../remoterepo
+  $ hg up -q master
+  $ echo a >> a
+  $ hg commit -m 'Move master bookmark'
+  $ cd ../localrepo
+  $ hg pull
+  pulling from ssh://user@dummy/remoterepo
+  searching for changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  (run 'hg update' to get a working copy)
+  $ hg bookmarks --remote
+     default/master            1:0238718db2b1
+
+Specify bookmark to pull
+  $ hg pull -B secondbook
+  pulling from ssh://user@dummy/remoterepo
+  searching for changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files (+1 heads)
+  (run 'hg heads' to see heads, 'hg merge' to merge)
+  $ hg bookmarks --remote
+     default/master            1:0238718db2b1
+     default/secondbook        2:ed7a9fd254d1
+
+Create second remote
+  $ cd ..
+  $ hg clone -q remoterepo secondremoterepo
+  $ cd secondremoterepo
+  $ hg up -q 0238718db2b1
+  $ hg book master
+  $ cd ..
+
+Add second remote repo path in localrepo
+  $ cd localrepo
+  $ cat >> $HGRCPATH << EOF
+  > [paths]
+  > secondremote=ssh://user@dummy/secondremoterepo
+  > EOF
+  $ hg pull secondremote
+  pulling from ssh://user@dummy/secondremoterepo
+  no changes found
+  $ hg book --remote
+     default/master            1:0238718db2b1
+     default/secondbook        2:ed7a9fd254d1
+     secondremote/master       1:0238718db2b1
+
+Move bookmark in second remote, pull and make sure it doesn't move in local repo
+  $ cd ../secondremoterepo
+  $ hg book secondbook
+  $ echo aaa >> a
+  $ hg commit -m 'Move bookmark in second remote'
+  $ cd ../localrepo
+  $ hg pull secondremote
+  pulling from ssh://user@dummy/secondremoterepo
+  no changes found
+
+Move bookmark in first remote, pull and make sure it moves in local repo
+  $ cd ../remoterepo
+  $ hg up -q secondbook
+  $ echo bbb > a
+  $ hg commit -m 'Moves second bookmark'
+  $ cd ../localrepo
+  $ hg pull
+  pulling from ssh://user@dummy/remoterepo
+  searching for changes
+  adding changesets
+  adding manifests
+  adding file changes
+  added 1 changesets with 1 changes to 1 files
+  (run 'hg update' to get a working copy)
+  $ hg bookmarks --remote
+     default/master            1:0238718db2b1
+     default/secondbook        3:c47dca9795c9
+     secondremote/master       1:0238718db2b1
+
+Delete bookmark on the server
+  $ cd ../remoterepo
+  $ hg book -d secondbook
+  $ cd ../localrepo
+  $ hg pull
+  pulling from ssh://user@dummy/remoterepo
+  no changes found
+  $ hg bookmarks --remote
+     default/master            1:0238718db2b1
+     secondremote/master       1:0238718db2b1