Patchwork [8,of,8,V2] cmdutil: apply dirstate.normallookup on (maybe partially) committed files

login
register
mail settings
Submitter Katsunori FUJIWARA
Date July 8, 2015, 8:08 a.m.
Message ID <17ca03748ee914f9696d.1436342924@feefifofum>
Download mbox | patch
Permalink /patch/9923/
State Accepted
Commit ff11c1565c04fbb178dd014336bef0b444fd9da4
Headers show

Comments

Katsunori FUJIWARA - July 8, 2015, 8:08 a.m.
# HG changeset patch
# User FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
# Date 1436342865 -32400
#      Wed Jul 08 17:07:45 2015 +0900
# Node ID 17ca03748ee914f9696dcec623bc2bc3977d8c38
# Parent  a3f47d16a56c9c64f45b928749de85386b8a5dc9
cmdutil: apply dirstate.normallookup on (maybe partially) committed files

To detect change of a file without redundant comparison of file
content, dirstate recognizes a file as certainly clean, if:

  (1) it is already known as "normal",
  (2) dirstate entry for it has valid (= not "-1") timestamp, and
  (3) mode, size and timestamp of it on the filesystem are as same as
      ones expected in dirstate

This works as expected in many cases, but doesn't in the corner case
that changing a file keeps mode, size and timestamp of it on the
filesystem.

The timetable below shows steps in one of typical such situations:

  ---- ----------------------------------- ----------------
                                           timestamp of "f"
                                           ----------------
                                           dirstate   file-
  time          action                     mem  file  system
  ---- ----------------------------------- ---- ----- -----
  N                                              ***   ***
       - change "f"                                     N

       - execute `hg commit -i`
         - backup "f" with timestamp N
         - revert "f" by `merge.update()`               N
           with `partially`
         - apply selected hunks                         N
           by `patch.patch()`

         - `repo.commit()`
           - `dirstate.normal("f")`         N
  N+1
           - `dirstate.write()`             N    N

         - restore "f"                                  N+1
         - restore timestamp of "f"                     N

       - `hg status` shows "f" as "clean"   N    N      N
  ---- ----------------------------------- ---- ----- -----

The most important point is that `dirstate.write()` is executed at N+1
or later. This causes writing dirstate timestamp N of "f" out
successfully. If it is executed at N, `parsers.pack_dirstate()`
replaces timestamp N with "-1" before actual writing dirstate out.

This issue can occur when `hg commit -i` satisfies conditions below:

  - the file is committed partially, and
  - mode and size of the file aren't changed before and after committing

The root cause of this issue is that (maybe partially changed) files
are restored with original timestamp but dirstate isn't updated for
them.

To detect changes of files correctly, this patch applies
`dirstate.normallookup()` on restored files. Status check is needed
before `dirstate.normallookup()`, because status other than "n(ormal)"
should be kept at failure of committing.

This patch doesn't examine whether each files are committed fully or
partially, because interactive hunk selection makes it difficult.

After this change, timetable is changed as below:

  ---- ----------------------------------- ----------------
                                           timestamp of "f"
                                           ----------------
                                           dirstate   file-
  time          action                     mem  file  system
  ---- ----------------------------------- ---- ----- -----
  N                                              ***   ***
       - change "f"                                     N

       - execute `hg commit -i`
         - backup "f" with timestamp N
         - revert "f" by `merge.update()`               N
           with `partially`
         - apply selected hunks                         N
           by `patch.internalpatch()`

         - `repo.commit()`
           - `dirstate.normal("f")`         N
  N+1
           - `dirstate.write()`             N    N

         - restore "f"                                  N+1
         - restore timestamp of "f"                     N
       ----------------------------------- ---- ----- -----
         - normallookup("f")                -1
         - release wlock
           - `dirstate.write()`             -1   -1     N
       ----------------------------------- ---- ----- -----

       - `hg status` shows "f" as "clean"   -1   -1     N
  ---- ----------------------------------- ---- ----- -----

To reproduce this issue in tests certainly, this patch emulates some
timing critical actions as below:

  - change "f" at N

    `touch -t 200001010000` before command invocation changes mtime of
    "f" to "2000-01-01 00:00" (= N).

  - apply selected hunks at N

    `patch.internalpatch()` with `fakepatchtime.py` explicitly changes
    mtime of patched files to "2000-01-01 00:00" (= N).

  - `dirstate.write()` at N+1 (or "not at N")

    `pack_dirstate()` uses actual timestamp at runtime as "now", and
    it should be different from the "2000-01-01 00:00" of "f".

BTW, in `test-commit-interactive.t`, files are sometimes treated as
modified , even though they are just committed fully via `hg commit
-i` and `hg diff` shows nothing for them.

Enabling win32text causes EOL style mismatching below:

  - files are changed in LF style EOL

    => files restored after committing uses LF style EOL (1)

  - `merge.update()` reverts files in CRLF style EOL
  - `patch.internalpatch()` changes files in CRLF style EOL

    => `dirstate.normal()` via `repo.commit()` uses the size of files
       in CRLF style EOL (2)

Therefore, fully committed files are treated as "modified", because
`lstat()` returns size of (1) restored files in LF style EOL, but
dirstate expects size of (2) committed files in CRLF style EOL.

After this patch, `dirstate.normallookup()` on committed files forces
subsequent `hg status` to examine changes exactly, and fully committed
files are treated as clean as expected.

This is reason why this patch also does:

  - add some `hg status` checking status of fully committed files
  - clear win32text configuration before size/timestamp sensitive examination
Martin von Zweigbergk - July 8, 2015, 8:28 p.m.
On Wed, Jul 8, 2015 at 1:15 AM FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
wrote:

> # HG changeset patch
> # User FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
> # Date 1436342865 -32400
> #      Wed Jul 08 17:07:45 2015 +0900
> # Node ID 17ca03748ee914f9696dcec623bc2bc3977d8c38
> # Parent  a3f47d16a56c9c64f45b928749de85386b8a5dc9
> cmdutil: apply dirstate.normallookup on (maybe partially) committed files
>
> To detect change of a file without redundant comparison of file
> content, dirstate recognizes a file as certainly clean, if:
>
>   (1) it is already known as "normal",
>   (2) dirstate entry for it has valid (= not "-1") timestamp, and
>   (3) mode, size and timestamp of it on the filesystem are as same as
>       ones expected in dirstate
>
> This works as expected in many cases, but doesn't in the corner case
> that changing a file keeps mode, size and timestamp of it on the
> filesystem.
>
> The timetable below shows steps in one of typical such situations:
>
>   ---- ----------------------------------- ----------------
>                                            timestamp of "f"
>                                            ----------------
>                                            dirstate   file-
>   time          action                     mem  file  system
>   ---- ----------------------------------- ---- ----- -----
>   N                                              ***   ***
>        - change "f"                                     N
>
>        - execute `hg commit -i`
>          - backup "f" with timestamp N
>          - revert "f" by `merge.update()`               N
>            with `partially`
>          - apply selected hunks                         N
>            by `patch.patch()`
>
>          - `repo.commit()`
>            - `dirstate.normal("f")`         N
>   N+1
>            - `dirstate.write()`             N    N
>
>          - restore "f"                                  N+1
>          - restore timestamp of "f"                     N
>
>        - `hg status` shows "f" as "clean"   N    N      N
>   ---- ----------------------------------- ---- ----- -----
>
> The most important point is that `dirstate.write()` is executed at N+1
> or later. This causes writing dirstate timestamp N of "f" out
> successfully. If it is executed at N, `parsers.pack_dirstate()`
> replaces timestamp N with "-1" before actual writing dirstate out.
>
> This issue can occur when `hg commit -i` satisfies conditions below:
>
>   - the file is committed partially, and
>   - mode and size of the file aren't changed before and after committing
>
> The root cause of this issue is that (maybe partially changed) files
> are restored with original timestamp but dirstate isn't updated for
> them.
>
> To detect changes of files correctly, this patch applies
> `dirstate.normallookup()` on restored files. Status check is needed
> before `dirstate.normallookup()`, because status other than "n(ormal)"
> should be kept at failure of committing.
>
> This patch doesn't examine whether each files are committed fully or
> partially, because interactive hunk selection makes it difficult.
>
> After this change, timetable is changed as below:
>
>   ---- ----------------------------------- ----------------
>                                            timestamp of "f"
>                                            ----------------
>                                            dirstate   file-
>   time          action                     mem  file  system
>   ---- ----------------------------------- ---- ----- -----
>   N                                              ***   ***
>        - change "f"                                     N
>
>        - execute `hg commit -i`
>          - backup "f" with timestamp N
>          - revert "f" by `merge.update()`               N
>            with `partially`
>          - apply selected hunks                         N
>            by `patch.internalpatch()`
>
>          - `repo.commit()`
>            - `dirstate.normal("f")`         N
>   N+1
>            - `dirstate.write()`             N    N
>
>          - restore "f"                                  N+1
>          - restore timestamp of "f"                     N
>        ----------------------------------- ---- ----- -----
>          - normallookup("f")                -1
>          - release wlock
>            - `dirstate.write()`             -1   -1     N
>        ----------------------------------- ---- ----- -----
>
>        - `hg status` shows "f" as "clean"   -1   -1     N
>   ---- ----------------------------------- ---- ----- -----
>
> To reproduce this issue in tests certainly, this patch emulates some
> timing critical actions as below:
>
>   - change "f" at N
>
>     `touch -t 200001010000` before command invocation changes mtime of
>     "f" to "2000-01-01 00:00" (= N).
>
>   - apply selected hunks at N
>
>     `patch.internalpatch()` with `fakepatchtime.py` explicitly changes
>     mtime of patched files to "2000-01-01 00:00" (= N).
>
>   - `dirstate.write()` at N+1 (or "not at N")
>
>     `pack_dirstate()` uses actual timestamp at runtime as "now", and
>     it should be different from the "2000-01-01 00:00" of "f".
>
> BTW, in `test-commit-interactive.t`, files are sometimes treated as
> modified , even though they are just committed fully via `hg commit
> -i` and `hg diff` shows nothing for them.
>
> Enabling win32text causes EOL style mismatching below:
>
>   - files are changed in LF style EOL
>
>     => files restored after committing uses LF style EOL (1)
>
>   - `merge.update()` reverts files in CRLF style EOL
>   - `patch.internalpatch()` changes files in CRLF style EOL
>
>     => `dirstate.normal()` via `repo.commit()` uses the size of files
>        in CRLF style EOL (2)
>
> Therefore, fully committed files are treated as "modified", because
> `lstat()` returns size of (1) restored files in LF style EOL, but
> dirstate expects size of (2) committed files in CRLF style EOL.
>
> After this patch, `dirstate.normallookup()` on committed files forces
> subsequent `hg status` to examine changes exactly, and fully committed
> files are treated as clean as expected.
>
> This is reason why this patch also does:
>
>   - add some `hg status` checking status of fully committed files
>   - clear win32text configuration before size/timestamp sensitive
> examination
>
> diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
> --- a/mercurial/cmdutil.py
> +++ b/mercurial/cmdutil.py
> @@ -203,8 +203,16 @@
>          finally:
>              # 5. finally restore backed-up files
>              try:
> +                dirstate = repo.dirstate
>                  for realname, tmpname in backups.iteritems():
>                      ui.debug('restoring %r to %r\n' % (tmpname, realname))
> +
> +                    if dirstate[realname] == 'n':
> +                        # without normallookup, restoring timestamp
> +                        # may cause that partially committed files
> +                        # aren't treated as modified
> +                        dirstate.normallookup(realname)
> +
>

Should this be done after writing the file as we do elsewhere? Maybe it
doesn't matter since the file will get an old timestamp anyway (because of
shutil.copystat)?

Besides that question, this series looks good to me. I haven't reviewed all
of it, but I like the new direction of it.


>                      util.copyfile(tmpname, repo.wjoin(realname))
>                      # Our calls to copystat() here and above are a
>                      # hack to trick any editors that have f open that
> diff --git a/tests/test-commit-interactive.t
> b/tests/test-commit-interactive.t
> --- a/tests/test-commit-interactive.t
> +++ b/tests/test-commit-interactive.t
> @@ -1381,6 +1381,8 @@
>    record this change to 'subdir/f1'? [Ynesfdaq?] y
>
>
> +  $ hg status -A subdir/f1
> +  C subdir/f1
>    $ hg tip -p
>    changeset:   28:* (glob)
>    tag:         tip
> @@ -1417,6 +1419,8 @@
>    +e
>    record this change to 'subdir/f1'? [Ynesfdaq?] y
>
> +  $ hg status -A subdir/f1
> +  C subdir/f1
>    $ hg log --template '{author}\n' -l 1
>    xyz
>    $ HGUSER="test"
> @@ -1426,7 +1430,7 @@
>  Moving files
>
>    $ hg update -C .
> -  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
> +  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
>    $ hg mv plain plain3
>    $ echo somechange >> plain3
>    $ hg commit -i -d '23 0' -mmoving_files << EOF
> @@ -1447,6 +1451,8 @@
>    record this change to 'plain3'? [Ynesfdaq?] y
>
>  The #if execbit block above changes the hash here on some systems
> +  $ hg status -A plain3
> +  C plain3
>    $ hg tip
>    changeset:   30:* (glob)
>    tag:         tip
> @@ -1526,3 +1532,98 @@
>    +foo
>
>    $ cd ..
> +
> +  $ hg status -A folder/bar
> +  C folder/bar
> +
> +Clear win32text configuration before size/timestamp sensitive test
> +
> +  $ cat >> .hg/hgrc <<EOF
> +  > [extensions]
> +  > win32text = !
> +  > [decode]
> +  > ** = !
> +  > [encode]
> +  > ** = !
> +  > [patch]
> +  > eol = strict
> +  > EOF
> +  $ hg update -q -C null
> +  $ hg update -q -C tip
> +
> +Test that partially committed file is still treated as "modified",
> +even if none of mode, size and timestamp is changed on the filesystem
> +(see also issue4583).
> +
> +  $ cat > subdir/f1 <<EOF
> +  > A
> +  > a
> +  > a
> +  > b
> +  > c
> +  > d
> +  > E
> +  > EOF
> +  $ hg diff --git subdir/f1
> +  diff --git a/subdir/f1 b/subdir/f1
> +  --- a/subdir/f1
> +  +++ b/subdir/f1
> +  @@ -1,7 +1,7 @@
> +  -a
> +  +A
> +   a
> +   a
> +   b
> +   c
> +   d
> +  -e
> +  +E
> +
> +  $ touch -t 200001010000 subdir/f1
> +
> +  $ cat >> .hg/hgrc <<EOF
> +  > # emulate invoking patch.internalpatch() at 2000-01-01 00:00
> +  > [fakepatchtime]
> +  > fakenow = 200001010000
> +  >
> +  > [extensions]
> +  > fakepatchtime = $TESTDIR/fakepatchtime.py
> +  > EOF
> +  $ hg commit -i -m 'commit subdir/f1 partially' <<EOF
> +  > y
> +  > y
> +  > n
> +  > EOF
> +  diff --git a/subdir/f1 b/subdir/f1
> +  2 hunks, 2 lines changed
> +  examine changes to 'subdir/f1'? [Ynesfdaq?] y
> +
> +  @@ -1,6 +1,6 @@
> +  -a
> +  +A
> +   a
> +   a
> +   b
> +   c
> +   d
> +  record change 1/2 to 'subdir/f1'? [Ynesfdaq?] y
> +
> +  @@ -2,6 +2,6 @@
> +   a
> +   a
> +   b
> +   c
> +   d
> +  -e
> +  +E
> +  record change 2/2 to 'subdir/f1'? [Ynesfdaq?] n
> +
> +  $ cat >> .hg/hgrc <<EOF
> +  > [extensions]
> +  > fakepatchtime = !
> +  > EOF
> +
> +  $ hg debugstate | grep ' subdir/f1$'
> +  n   0         -1 unset               subdir/f1
> +  $ hg status -A subdir/f1
> +  M subdir/f1
> _______________________________________________
> Mercurial-devel mailing list
> Mercurial-devel@selenic.com
> https://selenic.com/mailman/listinfo/mercurial-devel
>
Matt Mackall - July 8, 2015, 9:19 p.m.
On Wed, 2015-07-08 at 17:08 +0900, FUJIWARA Katsunori wrote:
> # HG changeset patch
> # User FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
> # Date 1436342865 -32400
> #      Wed Jul 08 17:07:45 2015 +0900
> # Node ID 17ca03748ee914f9696dcec623bc2bc3977d8c38
> # Parent  a3f47d16a56c9c64f45b928749de85386b8a5dc9
> cmdutil: apply dirstate.normallookup on (maybe partially) committed files

I've queued these for default, thanks. Note that I've done s/`/'/g on
your commit descriptions; it took me months to convince Pierre-Yves to
stop using these crazy quotes, please don't copy him!

Patch 8 looks correct to my eye, but it's possible I've missed
something.
Katsunori FUJIWARA - July 9, 2015, 3:45 p.m.
At Wed, 08 Jul 2015 20:28:36 +0000,
Martin von Zweigbergk wrote:

> On Wed, Jul 8, 2015 at 1:15 AM FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
> wrote:
> 
> > # HG changeset patch
> > # User FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
> > # Date 1436342865 -32400
> > #      Wed Jul 08 17:07:45 2015 +0900
> > # Node ID 17ca03748ee914f9696dcec623bc2bc3977d8c38
> > # Parent  a3f47d16a56c9c64f45b928749de85386b8a5dc9
> > cmdutil: apply dirstate.normallookup on (maybe partially) committed files
> >
> > To detect change of a file without redundant comparison of file
> > content, dirstate recognizes a file as certainly clean, if:
> >
> >   (1) it is already known as "normal",
> >   (2) dirstate entry for it has valid (= not "-1") timestamp, and
> >   (3) mode, size and timestamp of it on the filesystem are as same as
> >       ones expected in dirstate
> >
> > This works as expected in many cases, but doesn't in the corner case
> > that changing a file keeps mode, size and timestamp of it on the
> > filesystem.
> >
> > The timetable below shows steps in one of typical such situations:
> >
> >   ---- ----------------------------------- ----------------
> >                                            timestamp of "f"
> >                                            ----------------
> >                                            dirstate   file-
> >   time          action                     mem  file  system
> >   ---- ----------------------------------- ---- ----- -----
> >   N                                              ***   ***
> >        - change "f"                                     N
> >
> >        - execute `hg commit -i`
> >          - backup "f" with timestamp N
> >          - revert "f" by `merge.update()`               N
> >            with `partially`
> >          - apply selected hunks                         N
> >            by `patch.patch()`
> >
> >          - `repo.commit()`
> >            - `dirstate.normal("f")`         N
> >   N+1
> >            - `dirstate.write()`             N    N
> >
> >          - restore "f"                                  N+1
> >          - restore timestamp of "f"                     N
> >
> >        - `hg status` shows "f" as "clean"   N    N      N
> >   ---- ----------------------------------- ---- ----- -----
> >
> > The most important point is that `dirstate.write()` is executed at N+1
> > or later. This causes writing dirstate timestamp N of "f" out
> > successfully. If it is executed at N, `parsers.pack_dirstate()`
> > replaces timestamp N with "-1" before actual writing dirstate out.
> >
> > This issue can occur when `hg commit -i` satisfies conditions below:
> >
> >   - the file is committed partially, and
> >   - mode and size of the file aren't changed before and after committing
> >
> > The root cause of this issue is that (maybe partially changed) files
> > are restored with original timestamp but dirstate isn't updated for
> > them.
> >
> > To detect changes of files correctly, this patch applies
> > `dirstate.normallookup()` on restored files. Status check is needed
> > before `dirstate.normallookup()`, because status other than "n(ormal)"
> > should be kept at failure of committing.
> >
> > This patch doesn't examine whether each files are committed fully or
> > partially, because interactive hunk selection makes it difficult.
> >
> > After this change, timetable is changed as below:
> >
> >   ---- ----------------------------------- ----------------
> >                                            timestamp of "f"
> >                                            ----------------
> >                                            dirstate   file-
> >   time          action                     mem  file  system
> >   ---- ----------------------------------- ---- ----- -----
> >   N                                              ***   ***
> >        - change "f"                                     N
> >
> >        - execute `hg commit -i`
> >          - backup "f" with timestamp N
> >          - revert "f" by `merge.update()`               N
> >            with `partially`
> >          - apply selected hunks                         N
> >            by `patch.internalpatch()`
> >
> >          - `repo.commit()`
> >            - `dirstate.normal("f")`         N
> >   N+1
> >            - `dirstate.write()`             N    N
> >
> >          - restore "f"                                  N+1
> >          - restore timestamp of "f"                     N
> >        ----------------------------------- ---- ----- -----
> >          - normallookup("f")                -1
> >          - release wlock
> >            - `dirstate.write()`             -1   -1     N
> >        ----------------------------------- ---- ----- -----
> >
> >        - `hg status` shows "f" as "clean"   -1   -1     N
> >   ---- ----------------------------------- ---- ----- -----

(snip)

> >
> > diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
> > --- a/mercurial/cmdutil.py
> > +++ b/mercurial/cmdutil.py
> > @@ -203,8 +203,16 @@
> >          finally:
> >              # 5. finally restore backed-up files
> >              try:
> > +                dirstate = repo.dirstate
> >                  for realname, tmpname in backups.iteritems():
> >                      ui.debug('restoring %r to %r\n' % (tmpname, realname))
> > +
> > +                    if dirstate[realname] == 'n':
> > +                        # without normallookup, restoring timestamp
> > +                        # may cause that partially committed files
> > +                        # aren't treated as modified
> > +                        dirstate.normallookup(realname)
> > +
> >
> 
> Should this be done after writing the file as we do elsewhere? Maybe it
> doesn't matter since the file will get an old timestamp anyway (because of
> shutil.copystat)?

Yes, exactly speaking, 'lookupnormal()' should be invoked after
writing the file out.

On the other hand, IMHO, splitting below code block to place
'lookupnormal()' after 'util.copyfile()' seemed to weaken logical
meaning "restore file with original timestamp" of it.

                    util.copyfile(tmpname, repo.wjoin(realname))
                    # Our calls to copystat() here and above are a
                    # hack to trick any editors that have f open that
                    # we haven't modified them.
                    #
                    # Also note that this racy as an editor could
                    # notice the file's mtime before we've finished
                    # writing it.
                    shutil.copystat(tmpname, repo.wjoin(realname))
                    os.unlink(tmpname)

Should I move 'lookupnormal()' invocation ?

----------------------------------------------------------------------
[FUJIWARA Katsunori]                             foozy@lares.dti.ne.jp
Martin von Zweigbergk - July 9, 2015, 4:58 p.m.
On Thu, Jul 9, 2015 at 8:45 AM FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
wrote:

> At Wed, 08 Jul 2015 20:28:36 +0000,
> Martin von Zweigbergk wrote:
>
> > On Wed, Jul 8, 2015 at 1:15 AM FUJIWARA Katsunori <foozy@lares.dti.ne.jp
> >
> > wrote:
> >
> > > # HG changeset patch
> > > # User FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
> > > # Date 1436342865 -32400
> > > #      Wed Jul 08 17:07:45 2015 +0900
> > > # Node ID 17ca03748ee914f9696dcec623bc2bc3977d8c38
> > > # Parent  a3f47d16a56c9c64f45b928749de85386b8a5dc9
> > > cmdutil: apply dirstate.normallookup on (maybe partially) committed
> files
> > >
> > > To detect change of a file without redundant comparison of file
> > > content, dirstate recognizes a file as certainly clean, if:
> > >
> > >   (1) it is already known as "normal",
> > >   (2) dirstate entry for it has valid (= not "-1") timestamp, and
> > >   (3) mode, size and timestamp of it on the filesystem are as same as
> > >       ones expected in dirstate
> > >
> > > This works as expected in many cases, but doesn't in the corner case
> > > that changing a file keeps mode, size and timestamp of it on the
> > > filesystem.
> > >
> > > The timetable below shows steps in one of typical such situations:
> > >
> > >   ---- ----------------------------------- ----------------
> > >                                            timestamp of "f"
> > >                                            ----------------
> > >                                            dirstate   file-
> > >   time          action                     mem  file  system
> > >   ---- ----------------------------------- ---- ----- -----
> > >   N                                              ***   ***
> > >        - change "f"                                     N
> > >
> > >        - execute `hg commit -i`
> > >          - backup "f" with timestamp N
> > >          - revert "f" by `merge.update()`               N
> > >            with `partially`
> > >          - apply selected hunks                         N
> > >            by `patch.patch()`
> > >
> > >          - `repo.commit()`
> > >            - `dirstate.normal("f")`         N
> > >   N+1
> > >            - `dirstate.write()`             N    N
> > >
> > >          - restore "f"                                  N+1
> > >          - restore timestamp of "f"                     N
> > >
> > >        - `hg status` shows "f" as "clean"   N    N      N
> > >   ---- ----------------------------------- ---- ----- -----
> > >
> > > The most important point is that `dirstate.write()` is executed at N+1
> > > or later. This causes writing dirstate timestamp N of "f" out
> > > successfully. If it is executed at N, `parsers.pack_dirstate()`
> > > replaces timestamp N with "-1" before actual writing dirstate out.
> > >
> > > This issue can occur when `hg commit -i` satisfies conditions below:
> > >
> > >   - the file is committed partially, and
> > >   - mode and size of the file aren't changed before and after
> committing
> > >
> > > The root cause of this issue is that (maybe partially changed) files
> > > are restored with original timestamp but dirstate isn't updated for
> > > them.
> > >
> > > To detect changes of files correctly, this patch applies
> > > `dirstate.normallookup()` on restored files. Status check is needed
> > > before `dirstate.normallookup()`, because status other than "n(ormal)"
> > > should be kept at failure of committing.
> > >
> > > This patch doesn't examine whether each files are committed fully or
> > > partially, because interactive hunk selection makes it difficult.
> > >
> > > After this change, timetable is changed as below:
> > >
> > >   ---- ----------------------------------- ----------------
> > >                                            timestamp of "f"
> > >                                            ----------------
> > >                                            dirstate   file-
> > >   time          action                     mem  file  system
> > >   ---- ----------------------------------- ---- ----- -----
> > >   N                                              ***   ***
> > >        - change "f"                                     N
> > >
> > >        - execute `hg commit -i`
> > >          - backup "f" with timestamp N
> > >          - revert "f" by `merge.update()`               N
> > >            with `partially`
> > >          - apply selected hunks                         N
> > >            by `patch.internalpatch()`
> > >
> > >          - `repo.commit()`
> > >            - `dirstate.normal("f")`         N
> > >   N+1
> > >            - `dirstate.write()`             N    N
> > >
> > >          - restore "f"                                  N+1
> > >          - restore timestamp of "f"                     N
> > >        ----------------------------------- ---- ----- -----
> > >          - normallookup("f")                -1
> > >          - release wlock
> > >            - `dirstate.write()`             -1   -1     N
> > >        ----------------------------------- ---- ----- -----
> > >
> > >        - `hg status` shows "f" as "clean"   -1   -1     N
> > >   ---- ----------------------------------- ---- ----- -----
>
> (snip)
>
> > >
> > > diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
> > > --- a/mercurial/cmdutil.py
> > > +++ b/mercurial/cmdutil.py
> > > @@ -203,8 +203,16 @@
> > >          finally:
> > >              # 5. finally restore backed-up files
> > >              try:
> > > +                dirstate = repo.dirstate
> > >                  for realname, tmpname in backups.iteritems():
> > >                      ui.debug('restoring %r to %r\n' % (tmpname,
> realname))
> > > +
> > > +                    if dirstate[realname] == 'n':
> > > +                        # without normallookup, restoring timestamp
> > > +                        # may cause that partially committed files
> > > +                        # aren't treated as modified
> > > +                        dirstate.normallookup(realname)
> > > +
> > >
> >
> > Should this be done after writing the file as we do elsewhere? Maybe it
> > doesn't matter since the file will get an old timestamp anyway (because
> of
> > shutil.copystat)?
>
> Yes, exactly speaking, 'lookupnormal()' should be invoked after
> writing the file out.
>

IIUC, the only effect it might have is when lookupnormal() would give it
timestamp N and the file got timestamp N+1, in which case the file would
have to be unnecessarily checked for dirtiness. But since the timestamp
gets replaced anyway, it doesn't matter. So leave it as is, I think that's
fine.

Btw, I suppose this hack (the existing copystat() hack) means that all
involved files would have to be checked for dirtiness.


>
> On the other hand, IMHO, splitting below code block to place
> 'lookupnormal()' after 'util.copyfile()' seemed to weaken logical
> meaning "restore file with original timestamp" of it.
>
>                     util.copyfile(tmpname, repo.wjoin(realname))
>                     # Our calls to copystat() here and above are a
>                     # hack to trick any editors that have f open that
>                     # we haven't modified them.
>                     #
>                     # Also note that this racy as an editor could
>                     # notice the file's mtime before we've finished
>                     # writing it.
>                     shutil.copystat(tmpname, repo.wjoin(realname))
>                     os.unlink(tmpname)
>
> Should I move 'lookupnormal()' invocation ?
>
> ----------------------------------------------------------------------
> [FUJIWARA Katsunori]                             foozy@lares.dti.ne.jp
>
Katsunori FUJIWARA - July 13, 2015, 8:37 a.m.
At Thu, 09 Jul 2015 16:58:52 +0000,
Martin von Zweigbergk wrote:
> 
> On Thu, Jul 9, 2015 at 8:45 AM FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
> wrote:
> 
> > At Wed, 08 Jul 2015 20:28:36 +0000,
> > Martin von Zweigbergk wrote:
> >
> > > On Wed, Jul 8, 2015 at 1:15 AM FUJIWARA Katsunori <foozy@lares.dti.ne.jp
> > >
> > > wrote:
> > >
> > > > # HG changeset patch
> > > > # User FUJIWARA Katsunori <foozy@lares.dti.ne.jp>
> > > > # Date 1436342865 -32400
> > > > #      Wed Jul 08 17:07:45 2015 +0900
> > > > # Node ID 17ca03748ee914f9696dcec623bc2bc3977d8c38
> > > > # Parent  a3f47d16a56c9c64f45b928749de85386b8a5dc9
> > > > cmdutil: apply dirstate.normallookup on (maybe partially) committed files
> > > >
> > > > To detect change of a file without redundant comparison of file
> > > > content, dirstate recognizes a file as certainly clean, if:
> > > >
> > > >   (1) it is already known as "normal",
> > > >   (2) dirstate entry for it has valid (= not "-1") timestamp, and
> > > >   (3) mode, size and timestamp of it on the filesystem are as same as
> > > >       ones expected in dirstate
> > > >
> > > > This works as expected in many cases, but doesn't in the corner case
> > > > that changing a file keeps mode, size and timestamp of it on the
> > > > filesystem.
> > > >
> > > > The timetable below shows steps in one of typical such situations:
> > > >
> > > >   ---- ----------------------------------- ----------------
> > > >                                            timestamp of "f"
> > > >                                            ----------------
> > > >                                            dirstate   file-
> > > >   time          action                     mem  file  system
> > > >   ---- ----------------------------------- ---- ----- -----
> > > >   N                                              ***   ***
> > > >        - change "f"                                     N
> > > >
> > > >        - execute `hg commit -i`
> > > >          - backup "f" with timestamp N
> > > >          - revert "f" by `merge.update()`               N
> > > >            with `partially`
> > > >          - apply selected hunks                         N
> > > >            by `patch.patch()`
> > > >
> > > >          - `repo.commit()`
> > > >            - `dirstate.normal("f")`         N
> > > >   N+1
> > > >            - `dirstate.write()`             N    N
> > > >
> > > >          - restore "f"                                  N+1
> > > >          - restore timestamp of "f"                     N
> > > >
> > > >        - `hg status` shows "f" as "clean"   N    N      N
> > > >   ---- ----------------------------------- ---- ----- -----
> > > >
> > > > The most important point is that `dirstate.write()` is executed at N+1
> > > > or later. This causes writing dirstate timestamp N of "f" out
> > > > successfully. If it is executed at N, `parsers.pack_dirstate()`
> > > > replaces timestamp N with "-1" before actual writing dirstate out.
> > > >
> > > > This issue can occur when `hg commit -i` satisfies conditions below:
> > > >
> > > >   - the file is committed partially, and
> > > >   - mode and size of the file aren't changed before and after committing
> > > >
> > > > The root cause of this issue is that (maybe partially changed) files
> > > > are restored with original timestamp but dirstate isn't updated for
> > > > them.
> > > >
> > > > To detect changes of files correctly, this patch applies
> > > > `dirstate.normallookup()` on restored files. Status check is needed
> > > > before `dirstate.normallookup()`, because status other than "n(ormal)"
> > > > should be kept at failure of committing.
> > > >
> > > > This patch doesn't examine whether each files are committed fully or
> > > > partially, because interactive hunk selection makes it difficult.
> > > >
> > > > After this change, timetable is changed as below:
> > > >
> > > >   ---- ----------------------------------- ----------------
> > > >                                            timestamp of "f"
> > > >                                            ----------------
> > > >                                            dirstate   file-
> > > >   time          action                     mem  file  system
> > > >   ---- ----------------------------------- ---- ----- -----
> > > >   N                                              ***   ***
> > > >        - change "f"                                     N
> > > >
> > > >        - execute `hg commit -i`
> > > >          - backup "f" with timestamp N
> > > >          - revert "f" by `merge.update()`               N
> > > >            with `partially`
> > > >          - apply selected hunks                         N
> > > >            by `patch.internalpatch()`
> > > >
> > > >          - `repo.commit()`
> > > >            - `dirstate.normal("f")`         N
> > > >   N+1
> > > >            - `dirstate.write()`             N    N
> > > >
> > > >          - restore "f"                                  N+1
> > > >          - restore timestamp of "f"                     N
> > > >        ----------------------------------- ---- ----- -----
> > > >          - normallookup("f")                -1
> > > >          - release wlock
> > > >            - `dirstate.write()`             -1   -1     N
> > > >        ----------------------------------- ---- ----- -----
> > > >
> > > >        - `hg status` shows "f" as "clean"   -1   -1     N
> > > >   ---- ----------------------------------- ---- ----- -----
> >
> > (snip)
> >
> > > >
> > > > diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
> > > > --- a/mercurial/cmdutil.py
> > > > +++ b/mercurial/cmdutil.py
> > > > @@ -203,8 +203,16 @@
> > > >          finally:
> > > >              # 5. finally restore backed-up files
> > > >              try:
> > > > +                dirstate = repo.dirstate
> > > >                  for realname, tmpname in backups.iteritems():
> > > >                      ui.debug('restoring %r to %r\n' % (tmpname, realname))
> > > > +
> > > > +                    if dirstate[realname] == 'n':
> > > > +                        # without normallookup, restoring timestamp
> > > > +                        # may cause that partially committed files
> > > > +                        # aren't treated as modified
> > > > +                        dirstate.normallookup(realname)
> > > > +
> > > >
> > >
> > > Should this be done after writing the file as we do elsewhere? Maybe it
> > > doesn't matter since the file will get an old timestamp anyway (because
> > of
> > > shutil.copystat)?
> >
> > Yes, exactly speaking, 'lookupnormal()' should be invoked after
> > writing the file out.
> >
> 
> IIUC, the only effect it might have is when lookupnormal() would give it
> timestamp N and the file got timestamp N+1, in which case the file would
> have to be unnecessarily checked for dirtiness. But since the timestamp
> gets replaced anyway, it doesn't matter. So leave it as is, I think that's
> fine.
> 
> Btw, I suppose this hack (the existing copystat() hack) means that all
> involved files would have to be checked for dirtiness.

Would you mean that dirtiness check itself shouldn't rely on timestamp ?
or mention only about files involved in 'dorecord()' ?

If the latter, there is the reason to omit 'normallookup()'-ing on
some files, as mentioned in patch description.

> > > > Status check is needed before `dirstate.normallookup()`,
> > > > because status other than "n(ormal)" should be kept at failure
> > > > of committing.

At the time to apply "normallookup()" on involved files in finally
clause:

  - if changes are committed successfully:

    Available statuses of involved files in this case are:

    - 'n': (part or all of) changes are committed, or
           initial revision of that file is committed

  - otherwise (= failure of committing):

    Available statuses of involved files in this case are:

    - 'n': modified, but not yet committed
    - 'm': merged, but not yet committed
    - 'a': added, but not yet committed

    'm' and 'a' status should be kept for subsequent re-committing (or
    reverting), and restoring timestamp of these files has no
    problematic effect.

    BTW, files in "r" status after interactive selection aren't listed
    up in 'backups'. This is also reason why '?' (= already removed)
    is ignored in the former case.

Therefore, this patch omits "normallookup()"-ing on files in statuses
other than "n(ormal)".


BTW, 'shutil.copystat()' is used only in 'cmdutil.dorecord()'
fortunately :-)


> > On the other hand, IMHO, splitting below code block to place
> > 'lookupnormal()' after 'util.copyfile()' seemed to weaken logical
> > meaning "restore file with original timestamp" of it.
> >
> >                     util.copyfile(tmpname, repo.wjoin(realname))
> >                     # Our calls to copystat() here and above are a
> >                     # hack to trick any editors that have f open that
> >                     # we haven't modified them.
> >                     #
> >                     # Also note that this racy as an editor could
> >                     # notice the file's mtime before we've finished
> >                     # writing it.
> >                     shutil.copystat(tmpname, repo.wjoin(realname))
> >                     os.unlink(tmpname)
> >
> > Should I move 'lookupnormal()' invocation ?
> >
> > ----------------------------------------------------------------------
> > [FUJIWARA Katsunori]                             foozy@lares.dti.ne.jp
> >

----------------------------------------------------------------------
[FUJIWARA Katsunori]                             foozy@lares.dti.ne.jp

Patch

diff --git a/mercurial/cmdutil.py b/mercurial/cmdutil.py
--- a/mercurial/cmdutil.py
+++ b/mercurial/cmdutil.py
@@ -203,8 +203,16 @@ 
         finally:
             # 5. finally restore backed-up files
             try:
+                dirstate = repo.dirstate
                 for realname, tmpname in backups.iteritems():
                     ui.debug('restoring %r to %r\n' % (tmpname, realname))
+
+                    if dirstate[realname] == 'n':
+                        # without normallookup, restoring timestamp
+                        # may cause that partially committed files
+                        # aren't treated as modified
+                        dirstate.normallookup(realname)
+
                     util.copyfile(tmpname, repo.wjoin(realname))
                     # Our calls to copystat() here and above are a
                     # hack to trick any editors that have f open that
diff --git a/tests/test-commit-interactive.t b/tests/test-commit-interactive.t
--- a/tests/test-commit-interactive.t
+++ b/tests/test-commit-interactive.t
@@ -1381,6 +1381,8 @@ 
   record this change to 'subdir/f1'? [Ynesfdaq?] y
   
 
+  $ hg status -A subdir/f1
+  C subdir/f1
   $ hg tip -p
   changeset:   28:* (glob)
   tag:         tip
@@ -1417,6 +1419,8 @@ 
   +e
   record this change to 'subdir/f1'? [Ynesfdaq?] y
   
+  $ hg status -A subdir/f1
+  C subdir/f1
   $ hg log --template '{author}\n' -l 1
   xyz
   $ HGUSER="test"
@@ -1426,7 +1430,7 @@ 
 Moving files
 
   $ hg update -C .
-  1 files updated, 0 files merged, 0 files removed, 0 files unresolved
+  0 files updated, 0 files merged, 0 files removed, 0 files unresolved
   $ hg mv plain plain3
   $ echo somechange >> plain3
   $ hg commit -i -d '23 0' -mmoving_files << EOF
@@ -1447,6 +1451,8 @@ 
   record this change to 'plain3'? [Ynesfdaq?] y
   
 The #if execbit block above changes the hash here on some systems
+  $ hg status -A plain3
+  C plain3
   $ hg tip
   changeset:   30:* (glob)
   tag:         tip
@@ -1526,3 +1532,98 @@ 
   +foo
   
   $ cd ..
+
+  $ hg status -A folder/bar
+  C folder/bar
+
+Clear win32text configuration before size/timestamp sensitive test
+
+  $ cat >> .hg/hgrc <<EOF
+  > [extensions]
+  > win32text = !
+  > [decode]
+  > ** = !
+  > [encode]
+  > ** = !
+  > [patch]
+  > eol = strict
+  > EOF
+  $ hg update -q -C null
+  $ hg update -q -C tip
+
+Test that partially committed file is still treated as "modified",
+even if none of mode, size and timestamp is changed on the filesystem
+(see also issue4583).
+
+  $ cat > subdir/f1 <<EOF
+  > A
+  > a
+  > a
+  > b
+  > c
+  > d
+  > E
+  > EOF
+  $ hg diff --git subdir/f1
+  diff --git a/subdir/f1 b/subdir/f1
+  --- a/subdir/f1
+  +++ b/subdir/f1
+  @@ -1,7 +1,7 @@
+  -a
+  +A
+   a
+   a
+   b
+   c
+   d
+  -e
+  +E
+
+  $ touch -t 200001010000 subdir/f1
+
+  $ cat >> .hg/hgrc <<EOF
+  > # emulate invoking patch.internalpatch() at 2000-01-01 00:00
+  > [fakepatchtime]
+  > fakenow = 200001010000
+  > 
+  > [extensions]
+  > fakepatchtime = $TESTDIR/fakepatchtime.py
+  > EOF
+  $ hg commit -i -m 'commit subdir/f1 partially' <<EOF
+  > y
+  > y
+  > n
+  > EOF
+  diff --git a/subdir/f1 b/subdir/f1
+  2 hunks, 2 lines changed
+  examine changes to 'subdir/f1'? [Ynesfdaq?] y
+  
+  @@ -1,6 +1,6 @@
+  -a
+  +A
+   a
+   a
+   b
+   c
+   d
+  record change 1/2 to 'subdir/f1'? [Ynesfdaq?] y
+  
+  @@ -2,6 +2,6 @@
+   a
+   a
+   b
+   c
+   d
+  -e
+  +E
+  record change 2/2 to 'subdir/f1'? [Ynesfdaq?] n
+  
+  $ cat >> .hg/hgrc <<EOF
+  > [extensions]
+  > fakepatchtime = !
+  > EOF
+
+  $ hg debugstate | grep ' subdir/f1$'
+  n   0         -1 unset               subdir/f1
+  $ hg status -A subdir/f1
+  M subdir/f1