Patchwork [5,of,5] convert: introduce --config convert.hg.full=1 for adding all files, not just changed

login
register
mail settings
Submitter Mads Kiilerich
Date May 19, 2014, 8:33 p.m.
Message ID <fbd0070089aef388ffb4.1400531613@mk-desktop>
Download mbox | patch
Permalink /patch/4819/
State Changes Requested
Headers show

Comments

Mads Kiilerich - May 19, 2014, 8:33 p.m.
# HG changeset patch
# User Mads Kiilerich <madski@unity3d.com>
# Date 1400530735 -7200
#      Mon May 19 22:18:55 2014 +0200
# Node ID fbd0070089aef388ffb4bbae6ac34ff2bc17cd3f
# Parent  c28cc8a6bd6226f0588365d5ae8e3d8a42cfa063
convert: introduce --config convert.hg.full=1 for adding all files, not just changed

_Should_ in most cases not make any difference but will make convert
significantly slower.

Can be used when using filemap to add files that haven't been modified but are
made visible by removing and exclude or adding and include.

Note: This feature do not (yet) remove files that are excluded or no longer are
included.
Augie Fackler - May 26, 2014, 4:30 p.m.
On Mon, May 19, 2014 at 10:33:33PM +0200, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich <madski@unity3d.com>
> # Date 1400530735 -7200
> #      Mon May 19 22:18:55 2014 +0200
> # Node ID fbd0070089aef388ffb4bbae6ac34ff2bc17cd3f
> # Parent  c28cc8a6bd6226f0588365d5ae8e3d8a42cfa063
> convert: introduce --config convert.hg.full=1 for adding all files, not just changed

I'm not sure what this means...

>
> _Should_ in most cases not make any difference but will make convert
> significantly slower.
>
> Can be used when using filemap to add files that haven't been modified but are
> made visible by removing and exclude or adding and include.

So...I think what this feature is useful for is to resync with an
incremental conversion after you'd been intentionally omitting part of
the tree and no longer wish to.

Is that right?

>
> Note: This feature do not (yet) remove files that are excluded or no longer are
> included.
>
> diff --git a/hgext/convert/hg.py b/hgext/convert/hg.py
> --- a/hgext/convert/hg.py
> +++ b/hgext/convert/hg.py
> @@ -270,6 +270,7 @@ class mercurial_source(converter_source)
>          self.ignoreerrors = ui.configbool('convert', 'hg.ignoreerrors', False)
>          self.ignored = set()
>          self.saverev = ui.configbool('convert', 'hg.saverev', False)
> +        self.full = ui.configbool('convert', 'hg.full', False)
>          try:
>              self.repo = hg.repository(self.ui, path)
>              # try to provoke an exception if this isn't really a hg
> @@ -346,14 +347,17 @@ class mercurial_source(converter_source)
>              # self.ignored
>              self.getcopies(ctx, parents, files)
>              return [(f, rev) for f in files if f not in self.ignored], {}
> -        if self._changescache and self._changescache[0] == rev:
> +        if (self._changescache and self._changescache[0] == rev and
> +                not self.full):
>              m, a, r = self._changescache[1]
> +            d = u = i = c = []
>          else:
> -            m, a, r = self.repo.status(parents[0].node(), ctx.node())[:3]
> +            m, a, r, d, u, i, c = self.repo.status(parents[0].node(),
> +                                                   ctx.node(), clean=self.full)
>          # getcopies() detects missing revlogs early, run it before
>          # filtering the changes.
>          copies = self.getcopies(ctx, parents, m + a)
> -        changes = [(name, rev) for name in m + a + r
> +        changes = [(name, rev) for name in m + a + r + c
>                     if name not in self.ignored]
>          return sorted(changes), copies
>
> diff --git a/tests/test-convert-hg-sink.t b/tests/test-convert-hg-sink.t
> --- a/tests/test-convert-hg-sink.t
> +++ b/tests/test-convert-hg-sink.t
> @@ -515,11 +515,16 @@ An additional round, demonstrating that
>    o  0 0 (a-only f)
>
>
> -Conversion after rollback
> +Conversion after rollback and full conversion restoring unmasked but unmodified files
>
>    $ hg -R a rollback -f
>    repository tip rolled back to revision 2 (undo commit)
>
> +  $ cat >> $HGRCPATH <<EOF
> +  > [convert]
> +  > hg.full=True
> +  > EOF
> +
>    $ hg convert --filemap filemap-b 0 a --config convert.hg.revs=1::
>    scanning source...
>    sorting...
> @@ -527,7 +532,7 @@ Conversion after rollback
>    0 extra f+a-only change
>
>    $ hg -R a log -G -T '{rev} {desc|firstline} ({files})\n'
> -  o  3 extra f+a-only change (f)
> +  o  3 extra f+a-only change (b-only f)
>    |
>    o  2 x (f)
>    |
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@selenic.com
> http://selenic.com/mailman/listinfo/mercurial-devel
Mads Kiilerich - May 26, 2014, 5:26 p.m.
On 05/26/2014 06:30 PM, Augie Fackler wrote:
> On Mon, May 19, 2014 at 10:33:33PM +0200, Mads Kiilerich wrote:
>> # HG changeset patch
>> # User Mads Kiilerich <madski@unity3d.com>
>> # Date 1400530735 -7200
>> #      Mon May 19 22:18:55 2014 +0200
>> # Node ID fbd0070089aef388ffb4bbae6ac34ff2bc17cd3f
>> # Parent  c28cc8a6bd6226f0588365d5ae8e3d8a42cfa063
>> convert: introduce --config convert.hg.full=1 for adding all files, not just changed
> I'm not sure what this means...

It is hard to explain - especially in 80 characters and without making 
both the default and and this behaviour sound stupid.

The option should perhaps also be called something more verbose than 
just "full".

Suggestions would be appreciated.

>
>> _Should_ in most cases not make any difference but will make convert
>> significantly slower.
>>
>> Can be used when using filemap to add files that haven't been modified but are
>> made visible by removing and exclude or adding and include.
> So...I think what this feature is useful for is to resync with an
> incremental conversion after you'd been intentionally omitting part of
> the tree and no longer wish to.
>
> Is that right?

Yes. Except, in my case the main feature was that it made it possible to 
reintroduce files that unintentionally had been omitted ...

(Some bike shedding when looking for a better way to describe it:
"resync" is not a part of the existing convert lingo.
Referring to "incremental" in this way can also be confusing. It is more 
that setting this for an incremental conversion step can make it include 
files that didn't change in the revision you are "syncing" with.)

/Mads

>
>> Note: This feature do not (yet) remove files that are excluded or no longer are
>> included.
>>
>> diff --git a/hgext/convert/hg.py b/hgext/convert/hg.py
>> --- a/hgext/convert/hg.py
>> +++ b/hgext/convert/hg.py
>> @@ -270,6 +270,7 @@ class mercurial_source(converter_source)
>>           self.ignoreerrors = ui.configbool('convert', 'hg.ignoreerrors', False)
>>           self.ignored = set()
>>           self.saverev = ui.configbool('convert', 'hg.saverev', False)
>> +        self.full = ui.configbool('convert', 'hg.full', False)
>>           try:
>>               self.repo = hg.repository(self.ui, path)
>>               # try to provoke an exception if this isn't really a hg
>> @@ -346,14 +347,17 @@ class mercurial_source(converter_source)
>>               # self.ignored
>>               self.getcopies(ctx, parents, files)
>>               return [(f, rev) for f in files if f not in self.ignored], {}
>> -        if self._changescache and self._changescache[0] == rev:
>> +        if (self._changescache and self._changescache[0] == rev and
>> +                not self.full):
>>               m, a, r = self._changescache[1]
>> +            d = u = i = c = []
>>           else:
>> -            m, a, r = self.repo.status(parents[0].node(), ctx.node())[:3]
>> +            m, a, r, d, u, i, c = self.repo.status(parents[0].node(),
>> +                                                   ctx.node(), clean=self.full)
>>           # getcopies() detects missing revlogs early, run it before
>>           # filtering the changes.
>>           copies = self.getcopies(ctx, parents, m + a)
>> -        changes = [(name, rev) for name in m + a + r
>> +        changes = [(name, rev) for name in m + a + r + c
>>                      if name not in self.ignored]
>>           return sorted(changes), copies
>>
>> diff --git a/tests/test-convert-hg-sink.t b/tests/test-convert-hg-sink.t
>> --- a/tests/test-convert-hg-sink.t
>> +++ b/tests/test-convert-hg-sink.t
>> @@ -515,11 +515,16 @@ An additional round, demonstrating that
>>     o  0 0 (a-only f)
>>
>>
>> -Conversion after rollback
>> +Conversion after rollback and full conversion restoring unmasked but unmodified files
>>
>>     $ hg -R a rollback -f
>>     repository tip rolled back to revision 2 (undo commit)
>>
>> +  $ cat >> $HGRCPATH <<EOF
>> +  > [convert]
>> +  > hg.full=True
>> +  > EOF
>> +
>>     $ hg convert --filemap filemap-b 0 a --config convert.hg.revs=1::
>>     scanning source...
>>     sorting...
>> @@ -527,7 +532,7 @@ Conversion after rollback
>>     0 extra f+a-only change
>>
>>     $ hg -R a log -G -T '{rev} {desc|firstline} ({files})\n'
>> -  o  3 extra f+a-only change (f)
>> +  o  3 extra f+a-only change (b-only f)
>>     |
>>     o  2 x (f)
>>     |
>> _______________________________________________
>> Mercurial-devel mailing list
>> Mercurial-devel@selenic.com
>> http://selenic.com/mailman/listinfo/mercurial-devel
Matt Mackall - May 27, 2014, 1:46 a.m.
On Mon, 2014-05-26 at 19:26 +0200, Mads Kiilerich wrote:
> On 05/26/2014 06:30 PM, Augie Fackler wrote:
> > On Mon, May 19, 2014 at 10:33:33PM +0200, Mads Kiilerich wrote:
> >> # HG changeset patch
> >> # User Mads Kiilerich <madski@unity3d.com>
> >> # Date 1400530735 -7200
> >> #      Mon May 19 22:18:55 2014 +0200
> >> # Node ID fbd0070089aef388ffb4bbae6ac34ff2bc17cd3f
> >> # Parent  c28cc8a6bd6226f0588365d5ae8e3d8a42cfa063
> >> convert: introduce --config convert.hg.full=1 for adding all files, not just changed
> > I'm not sure what this means...
> 
> It is hard to explain - especially in 80 characters and without making 
> both the default and and this behaviour sound stupid.
> 
> The option should perhaps also be called something more verbose than 
> just "full".

Probably something like "checkunmodified". I presume this doesn't
actually commit new copies of files that are already tracked, right?
Mads Kiilerich - May 27, 2014, 11 a.m.
On 05/27/2014 03:46 AM, Matt Mackall wrote:
> On Mon, 2014-05-26 at 19:26 +0200, Mads Kiilerich wrote:
>> On 05/26/2014 06:30 PM, Augie Fackler wrote:
>>> On Mon, May 19, 2014 at 10:33:33PM +0200, Mads Kiilerich wrote:
>>>> # HG changeset patch
>>>> # User Mads Kiilerich <madski@unity3d.com>
>>>> # Date 1400530735 -7200
>>>> #      Mon May 19 22:18:55 2014 +0200
>>>> # Node ID fbd0070089aef388ffb4bbae6ac34ff2bc17cd3f
>>>> # Parent  c28cc8a6bd6226f0588365d5ae8e3d8a42cfa063
>>>> convert: introduce --config convert.hg.full=1 for adding all files, not just changed
>>> I'm not sure what this means...
>> It is hard to explain - especially in 80 characters and without making
>> both the default and and this behaviour sound stupid.
>>
>> The option should perhaps also be called something more verbose than
>> just "full".
> Probably something like "checkunmodified". I presume this doesn't
> actually commit new copies of files that are already tracked, right?

It will commit all files, but unmodified files will not create a new 
filelog entry and will thus be noops not creating new copies.

(I tried enabling it everywhere and investigated all the differences it 
gave. Everything (including hashes) stayed the same except where there 
was good reasons it made a difference.)

So it will actually not "check", even though the user experience is that 
it checks for missing files and then (more important) adds them.


It should perhaps also be named to make it clear that it is an option on 
the hg source, not the hg sink.

Anyway, the main point is that it will get all the files back in sync. 
So "convert.hg.syncallfiles" or "convert.hg.reconvertallfiles"?

/Mads

Patch

diff --git a/hgext/convert/hg.py b/hgext/convert/hg.py
--- a/hgext/convert/hg.py
+++ b/hgext/convert/hg.py
@@ -270,6 +270,7 @@  class mercurial_source(converter_source)
         self.ignoreerrors = ui.configbool('convert', 'hg.ignoreerrors', False)
         self.ignored = set()
         self.saverev = ui.configbool('convert', 'hg.saverev', False)
+        self.full = ui.configbool('convert', 'hg.full', False)
         try:
             self.repo = hg.repository(self.ui, path)
             # try to provoke an exception if this isn't really a hg
@@ -346,14 +347,17 @@  class mercurial_source(converter_source)
             # self.ignored
             self.getcopies(ctx, parents, files)
             return [(f, rev) for f in files if f not in self.ignored], {}
-        if self._changescache and self._changescache[0] == rev:
+        if (self._changescache and self._changescache[0] == rev and
+                not self.full):
             m, a, r = self._changescache[1]
+            d = u = i = c = []
         else:
-            m, a, r = self.repo.status(parents[0].node(), ctx.node())[:3]
+            m, a, r, d, u, i, c = self.repo.status(parents[0].node(),
+                                                   ctx.node(), clean=self.full)
         # getcopies() detects missing revlogs early, run it before
         # filtering the changes.
         copies = self.getcopies(ctx, parents, m + a)
-        changes = [(name, rev) for name in m + a + r
+        changes = [(name, rev) for name in m + a + r + c
                    if name not in self.ignored]
         return sorted(changes), copies
 
diff --git a/tests/test-convert-hg-sink.t b/tests/test-convert-hg-sink.t
--- a/tests/test-convert-hg-sink.t
+++ b/tests/test-convert-hg-sink.t
@@ -515,11 +515,16 @@  An additional round, demonstrating that 
   o  0 0 (a-only f)
   
 
-Conversion after rollback
+Conversion after rollback and full conversion restoring unmasked but unmodified files
 
   $ hg -R a rollback -f
   repository tip rolled back to revision 2 (undo commit)
 
+  $ cat >> $HGRCPATH <<EOF
+  > [convert]
+  > hg.full=True
+  > EOF
+
   $ hg convert --filemap filemap-b 0 a --config convert.hg.revs=1::
   scanning source...
   sorting...
@@ -527,7 +532,7 @@  Conversion after rollback
   0 extra f+a-only change
 
   $ hg -R a log -G -T '{rev} {desc|firstline} ({files})\n'
-  o  3 extra f+a-only change (f)
+  o  3 extra f+a-only change (b-only f)
   |
   o  2 x (f)
   |