Patchwork [3,of,7] share: add support "full share" suport

login
register
mail settings
Submitter Angel Ezquerra
Date Nov. 25, 2015, 12:22 a.m.
Message ID <fd4449f9e1239bc269e7.1448410947@waste.org>
Download mbox | patch
Permalink /patch/11633/
State Changes Requested
Headers show

Comments

Angel Ezquerra - Nov. 25, 2015, 12:22 a.m.
# HG changeset patch
# User Angel Ezquerra <angel.ezquerra@gmail.com>
# Date 1419360788 -3600
#      Tue Dec 23 19:53:08 2014 +0100
# Node ID fd4449f9e1239bc269e7a842168ace286ee4dd9e
# Parent  d7b08e9f7782fdc21663dff41149c054862276f5
share: add support "full share" suport

Add a new --full flag that makes it possible to create "full shares".

A "fully shared repository" is one shares _everything_ with its source
repository except for the working directory, its corresponding state (i.e.
parent revisions, update, merge, rebase and mq states) and its configuration
files.

Fully shared repositories share their bookmarks and mq patches. The only
difference between the source repo and the fully shared copy is that their
working directory can be different and point to a different revision, bookmark
and branch, and can be in a different merge, update, rebase, shelve, histedit
or mq state.

To implement fully shared repositories we replace the regular localrepo vfs
object with a remappedvfs object which points to the share source except for a
few key files (dirstate, branch, bookmarks.current, sharedpath, shared, hgrc
and some pending and undo files). The idea is to only keep the files that are
strictly necessary to maintain separate working directories in the shared
repository, and share everything else with the shared source repository.

Since the "non-store" parts of a "full repository share" are split between the
source repository (which contains most of the repository files) and the shared
repository itself (which contains the working directory state related files and
some config files), we must lock both repositories when locking the shared
repository. This is done by using a multilock that holds two locks, the source
repository wlock and shared repository wlock.

This revision does not add support for unsharing a fully shared repository. That
will be added on a later patch.

# NOTE:
- This is the first step on the "subrepo store" plan that we discussed during
the mercurial 3.2 and 3.6 sprints
Yuya Nishihara - Nov. 29, 2015, 2:42 p.m.
On Tue, 24 Nov 2015 18:22:27 -0600, Angel Ezquerra wrote:
> # HG changeset patch
> # User Angel Ezquerra <angel.ezquerra@gmail.com>
> # Date 1419360788 -3600
> #      Tue Dec 23 19:53:08 2014 +0100
> # Node ID fd4449f9e1239bc269e7a842168ace286ee4dd9e
> # Parent  d7b08e9f7782fdc21663dff41149c054862276f5
> share: add support "full share" suport

> --- a/mercurial/localrepo.py
> +++ b/mercurial/localrepo.py
> @@ -190,6 +190,27 @@
>      def changegroupsubset(self, bases, heads, source):
>          return changegroup.changegroupsubset(self._repo, bases, heads, source)
>  
> +# List of files that belong to the "local repository" rather than to the shared
> +# repository source when using shared repositories
> +# This list includes the files that belong to the "working directory state" plus
> +# some config files (i.e. the hgrc and the share config files)
> +# This list is used by to create "full repository shares"
> +# It is placed here so that it can be easily extended by external users (e.g.
> +# TortoiseHg, etc)
> +# Note that this list does not include some cache files that could be placed on
> +# the local repository to improve performance when using shared repos
> +LOCALREPOFILES = (

We generally use lowercase name.

> +    'hgrc', 'sharedpath', 'shared',
> +    'dirstate', 'branch', 'undo.dirstate', 'undo.branch',
> +    'dirstate.pending',
> +    'bookmarks.current',
> +    'last-message.txt',
> +    'updatestate', 'rebasestate', 'graftstate',
> +    'shelvedstate', 'unshelverebasestate',
> +    'histedit-state',
> +    'patches/status',
> +)

I guess this list is incomplete (e.g. missing bisect.state?) and the number
of items would burst.

> @@ -278,13 +300,36 @@
>  
>          self.sharedpath = self.path
>          try:
> -            vfs = scmutil.vfs(self.vfs.read("sharedpath").rstrip('\n'),
> -                              realpath=True)
> +            localpath = self.path
> +            def localjoinfn(p):
> +                return os.path.join(localpath, p)
> +            def joinfn(p):
> +                if not p:
> +                    return p
> +                # We don't share the journal
> +                if p in LOCALREPOFILES:
> +                    return localjoinfn(p)
> +                base = os.path.dirname(p)
> +                if base.endswith('/merge'):
> +                    return localjoinfn(p)

I didn't read it carefully, but base.endswith('/merge') never be true.

> -        l = self._lock(self.vfs, "wlock", wait, unlock,
> +        if self.fullshare:
> +            sourcelock = self._lock(self.vfs, "wlock", wait, None,
> +                       self.invalidatedirstate, _('share source of %s (%s)') %
> +                                    (self.origroot, self.sharedpath),
> +                       inheritchecker=self._wlockchecktransaction,
> +                       parentenvvar='HG_WLOCK_LOCKER')
> +            locallock = self._lock(self.localvfs, "wlock", wait, unlock,
> +                       self.invalidatedirstate, _('working directory of %s') %
> +                       self.origroot,
> +                       inheritchecker=self._wlockchecktransaction,
> +                       parentenvvar='HG_WLOCK_LOCKER')

If sourcelock can be taken but locallock fail, sourcelock won't be released
explicitly.

> +            l = lockmod.multilock(locallock, sourcelock)

Wrong order? I think you've designed the multilock to take locks in order,
and release them in reverse order.

> +        else:
> +            l = self._lock(self.vfs, "wlock", wait, unlock,
>                         self.invalidatedirstate, _('working directory of %s') %
>                         self.origroot,
>                         inheritchecker=self._wlockchecktransaction,

Patch

diff --git a/hgext/share.py b/hgext/share.py
--- a/hgext/share.py
+++ b/hgext/share.py
@@ -52,10 +52,11 @@ 
 
 @command('share',
     [('U', 'noupdate', None, _('do not create a working directory')),
-     ('B', 'bookmarks', None, _('also share bookmarks'))],
-    _('[-U] [-B] SOURCE [DEST]'),
+     ('B', 'bookmarks', None, _('also share bookmarks')),
+     ('', 'full', None, _('share in full'))],
+     _('[-U] [-B] [--full] SOURCE [DEST]'),
     norepo=True)
-def share(ui, source, dest=None, noupdate=False, bookmarks=False):
+def share(ui, source, dest=None, noupdate=False, bookmarks=False, full=False):
     """create a new shared repository
 
     Initialize a new repository and working directory that shares its
@@ -73,7 +74,7 @@ 
        the broken clone to reset it to a changeset that still exists.
     """
 
-    return hg.share(ui, source, dest, not noupdate, bookmarks)
+    return hg.share(ui, source, dest, not noupdate, bookmarks, fullshare=full)
 
 @command('unshare', [], '')
 def unshare(ui, repo):
@@ -82,8 +83,11 @@ 
     Copy the store data to the repo and remove the sharedpath data.
     """
 
-    if not repo.shared():
+    sharetype = repo.shared()
+    if not sharetype:
         raise error.Abort(_("this is not a shared repo"))
+    elif sharetype == 'all':
+        raise error.Abort(_("unsharing a fully shared repo is not supported"))
 
     destlock = lock = None
     lock = repo.lock()
diff --git a/mercurial/hg.py b/mercurial/hg.py
--- a/mercurial/hg.py
+++ b/mercurial/hg.py
@@ -190,11 +190,14 @@ 
         return ''
     return os.path.basename(os.path.normpath(path))
 
-def share(ui, source, dest=None, update=True, bookmarks=True):
+def share(ui, source, dest=None, update=True, bookmarks=True, fullshare=False):
     '''create a shared repository'''
 
     if not islocal(source):
         raise error.Abort(_('can only share local repositories'))
+    if bookmarks and fullshare:
+        ui.warn(_('full repository shares implicitly share bookmarks'))
+        bookmarks = False
 
     if not dest:
         dest = defaultdest(source)
@@ -261,6 +264,10 @@ 
         fp = r.vfs('shared', 'w')
         fp.write('bookmarks\n')
         fp.close()
+    if fullshare:
+        fp = r.vfs('shared', 'w')
+        fp.write('all\n')
+        fp.close()
 
 def copystore(ui, srcrepo, destpath):
     '''copy files from store of srcrepo in destpath
diff --git a/mercurial/localrepo.py b/mercurial/localrepo.py
--- a/mercurial/localrepo.py
+++ b/mercurial/localrepo.py
@@ -190,6 +190,27 @@ 
     def changegroupsubset(self, bases, heads, source):
         return changegroup.changegroupsubset(self._repo, bases, heads, source)
 
+# List of files that belong to the "local repository" rather than to the shared
+# repository source when using shared repositories
+# This list includes the files that belong to the "working directory state" plus
+# some config files (i.e. the hgrc and the share config files)
+# This list is used by to create "full repository shares"
+# It is placed here so that it can be easily extended by external users (e.g.
+# TortoiseHg, etc)
+# Note that this list does not include some cache files that could be placed on
+# the local repository to improve performance when using shared repos
+LOCALREPOFILES = (
+    'hgrc', 'sharedpath', 'shared',
+    'dirstate', 'branch', 'undo.dirstate', 'undo.branch',
+    'dirstate.pending',
+    'bookmarks.current',
+    'last-message.txt',
+    'updatestate', 'rebasestate', 'graftstate',
+    'shelvedstate', 'unshelverebasestate',
+    'histedit-state',
+    'patches/status',
+)
+
 class localrepository(object):
 
     supportedformats = set(('revlogv1', 'generaldelta', 'treemanifest',
@@ -208,6 +229,7 @@ 
 
     def __init__(self, baseui, path=None, create=False):
         self.requirements = set()
+        self.fullshare = False
         self.wvfs = scmutil.vfs(path, expandpath=True, realpath=True)
         self.wopener = self.wvfs
         self.root = self.wvfs.base
@@ -278,13 +300,36 @@ 
 
         self.sharedpath = self.path
         try:
-            vfs = scmutil.vfs(self.vfs.read("sharedpath").rstrip('\n'),
-                              realpath=True)
+            localpath = self.path
+            def localjoinfn(p):
+                return os.path.join(localpath, p)
+            def joinfn(p):
+                if not p:
+                    return p
+                # We don't share the journal
+                if p in LOCALREPOFILES:
+                    return localjoinfn(p)
+                base = os.path.dirname(p)
+                if base.endswith('/merge'):
+                    return localjoinfn(p)
+                return None
+            vfs = scmutil.remappedvfs(self.vfs.read("sharedpath").rstrip('\n'),
+                              realpath=True, joinfn=joinfn)
             s = vfs.base
             if not vfs.exists():
                 raise error.RepoError(
                     _('.hg/sharedpath points to nonexistent directory %s') % s)
             self.sharedpath = s
+            # is this a full share?
+            self.fullshare = self.vfs.read('shared') == 'all\n'
+            if self.fullshare:
+                # full shares use the remappedvfs created above to control
+                # which files are read from which repository
+                self.localpath = self.path
+                self.path = self.sharedpath
+                self.localvfs = self.vfs
+                self.vfs = vfs
+                self.opener = self.vfs
         except IOError as inst:
             if inst.errno != errno.ENOENT:
                 raise
@@ -825,7 +870,9 @@ 
 
     def shared(self):
         '''the type of shared repository (None if not shared)'''
-        if self.sharedpath != self.path:
+        if self.fullshare:
+            return 'all'
+        elif self.sharedpath != self.path:
             return 'store'
         return None
 
@@ -1325,7 +1372,20 @@ 
 
             self._filecache['dirstate'].refresh()
 
-        l = self._lock(self.vfs, "wlock", wait, unlock,
+        if self.fullshare:
+            sourcelock = self._lock(self.vfs, "wlock", wait, None,
+                       self.invalidatedirstate, _('share source of %s (%s)') %
+                                    (self.origroot, self.sharedpath),
+                       inheritchecker=self._wlockchecktransaction,
+                       parentenvvar='HG_WLOCK_LOCKER')
+            locallock = self._lock(self.localvfs, "wlock", wait, unlock,
+                       self.invalidatedirstate, _('working directory of %s') %
+                       self.origroot,
+                       inheritchecker=self._wlockchecktransaction,
+                       parentenvvar='HG_WLOCK_LOCKER')
+            l = lockmod.multilock(locallock, sourcelock)
+        else:
+            l = self._lock(self.vfs, "wlock", wait, unlock,
                        self.invalidatedirstate, _('working directory of %s') %
                        self.origroot,
                        inheritchecker=self._wlockchecktransaction,
diff --git a/tests/test-share.t b/tests/test-share.t
--- a/tests/test-share.t
+++ b/tests/test-share.t
@@ -297,6 +297,236 @@ 
      bm4                       5:92793bfc8cad
   $ cd ..
 
+create a full share
+
+  $ hg share --full repo1 repo-fullshare
+  updating working directory
+  2 files updated, 0 files merged, 0 files removed, 0 files unresolved
+
+  $ cd repo1
+  $ hg summary
+  parent: 3:b87954705719 
+   testing shared bookmarks
+  branch: default
+  bookmarks: *bm1
+  commit: (clean)
+  update: 2 new changesets, 2 branch heads (merge)
+  $ hg log -G
+  o  changeset:   5:92793bfc8cad
+  |  bookmark:    bm4
+  |  tag:         tip
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     foo in b
+  |
+  o  changeset:   4:62f4ded848e4
+  |  bookmark:    bm3
+  |  parent:      2:c2e0ac586386
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     testing shared bookmarks
+  |
+  | @  changeset:   3:b87954705719
+  |/   bookmark:    bm1
+  |    user:        test
+  |    date:        Thu Jan 01 00:00:00 1970 +0000
+  |    summary:     testing shared bookmarks
+  |
+  o  changeset:   2:c2e0ac586386
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     another file
+  |
+  o  changeset:   1:8af4dc49db9e
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     change in shared clone
+  |
+  o  changeset:   0:d3873e73d99e
+     user:        test
+     date:        Thu Jan 01 00:00:00 1970 +0000
+     summary:     init
+  
+  $ cd ..
+  $ cd repo-fullshare
+
+full shares share the whole set of revisions, but not the current revision
+
+  $ hg summary
+  parent: 5:92793bfc8cad tip
+   foo in b
+  branch: default
+  bookmarks: bm4
+  commit: (clean)
+  update: 1 new changesets, 2 branch heads (merge)
+
+  $ hg log -G
+  @  changeset:   5:92793bfc8cad
+  |  bookmark:    bm4
+  |  tag:         tip
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     foo in b
+  |
+  o  changeset:   4:62f4ded848e4
+  |  bookmark:    bm3
+  |  parent:      2:c2e0ac586386
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     testing shared bookmarks
+  |
+  | o  changeset:   3:b87954705719
+  |/   bookmark:    bm1
+  |    user:        test
+  |    date:        Thu Jan 01 00:00:00 1970 +0000
+  |    summary:     testing shared bookmarks
+  |
+  o  changeset:   2:c2e0ac586386
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     another file
+  |
+  o  changeset:   1:8af4dc49db9e
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     change in shared clone
+  |
+  o  changeset:   0:d3873e73d99e
+     user:        test
+     date:        Thu Jan 01 00:00:00 1970 +0000
+     summary:     init
+  
+full shares share their bookmarks but not the active bookmark
+  $ hg bookmark bm5
+  $ hg bookmarks
+     bm1                       3:b87954705719
+     bm3                       4:62f4ded848e4
+     bm4                       5:92793bfc8cad
+   * bm5                       5:92793bfc8cad
+  $ hg -R ../repo1 bookmarks
+   * bm1                       3:b87954705719
+     bm3                       4:62f4ded848e4
+     bm4                       5:92793bfc8cad
+     bm5                       5:92793bfc8cad
+
+the source and shared repo working directories can point to different revisions
+and bookmarks
+  $ hg update bm3
+  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  (activating bookmark bm3)
+  $ hg log -r .
+  changeset:   4:62f4ded848e4
+  bookmark:    bm3
+  parent:      2:c2e0ac586386
+  user:        test
+  date:        Thu Jan 01 00:00:00 1970 +0000
+  summary:     testing shared bookmarks
+  
+  $ hg -R ../repo1 log -r .
+  changeset:   3:b87954705719
+  bookmark:    bm1
+  user:        test
+  date:        Thu Jan 01 00:00:00 1970 +0000
+  summary:     testing shared bookmarks
+  
+they can also commit to different branches
+  $ hg branch createdinshare
+  marked working directory as branch createdinshare
+  (branches are permanent and global, did you want a bookmark?)
+  $ hg branch
+  createdinshare
+  $ hg -R ../repo1 branch
+  default
+
+it is possible to commit on top of different revisions
+  $ cat a
+  more shared bookmarks
+  $ echo a > a
+  $ hg commit -m "created in full share"
+  $ echo a > ../repo1/a
+  $ hg commit -R ../repo1 -m "created in share source"
+  $ hg log -G
+  o  changeset:   7:e0fb5f85a10b
+  |  bookmark:    bm1
+  |  tag:         tip
+  |  parent:      3:b87954705719
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     created in share source
+  |
+  | @  changeset:   6:5ea6503932c4
+  | |  branch:      createdinshare
+  | |  bookmark:    bm3
+  | |  parent:      4:62f4ded848e4
+  | |  user:        test
+  | |  date:        Thu Jan 01 00:00:00 1970 +0000
+  | |  summary:     created in full share
+  | |
+  | | o  changeset:   5:92793bfc8cad
+  | |/   bookmark:    bm4
+  | |    bookmark:    bm5
+  | |    user:        test
+  | |    date:        Thu Jan 01 00:00:00 1970 +0000
+  | |    summary:     foo in b
+  | |
+  | o  changeset:   4:62f4ded848e4
+  | |  parent:      2:c2e0ac586386
+  | |  user:        test
+  | |  date:        Thu Jan 01 00:00:00 1970 +0000
+  | |  summary:     testing shared bookmarks
+  | |
+  o |  changeset:   3:b87954705719
+  |/   user:        test
+  |    date:        Thu Jan 01 00:00:00 1970 +0000
+  |    summary:     testing shared bookmarks
+  |
+  o  changeset:   2:c2e0ac586386
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     another file
+  |
+  o  changeset:   1:8af4dc49db9e
+  |  user:        test
+  |  date:        Thu Jan 01 00:00:00 1970 +0000
+  |  summary:     change in shared clone
+  |
+  o  changeset:   0:d3873e73d99e
+     user:        test
+     date:        Thu Jan 01 00:00:00 1970 +0000
+     summary:     init
+  
+
+full shares should have a limited set of files in their .hg folder
+
+  $ ls .hg
+  bookmarks.current
+  branch
+  cache
+  dirstate
+  last-message.txt
+  requires
+  shared
+  sharedpath
+  undo.branch
+  undo.dirstate
+
+  $ ls ../repo1/.hg
+  00changelog.i
+  bookmarks
+  bookmarks.current
+  branch
+  cache
+  dirstate
+  last-message.txt
+  requires
+  store
+  undo.backup.bookmarks
+  undo.backup.dirstate
+  undo.bookmarks
+  undo.branch
+  undo.desc
+  undo.dirstate
+
 Explicitly kill daemons to let the test exit on Windows
 
   $ killdaemons.py