Patchwork [2,of,4] killdaemons: close pid file before killing processes

login
register
mail settings
Submitter Matt Harbison
Date June 5, 2017, 3:38 a.m.
Message ID <eaed97a21942af11c122.1496633929@Envy>
Download mbox | patch
Permalink /patch/21202/
State Accepted
Headers show

Comments

Matt Harbison - June 5, 2017, 3:38 a.m.
# HG changeset patch
# User Matt Harbison <matt_harbison@yahoo.com>
# Date 1495503902 14400
#      Mon May 22 21:45:02 2017 -0400
# Node ID eaed97a21942af11c122607bf37f7399c68cae9d
# Parent  ecc27f3123ea173f2dc66e20abbedad5741ea5e1
killdaemons: close pid file before killing processes

With #serve enabled on Windows, I was getting occasional stacktraces like this:

  Errored test-hgweb-json.t: Traceback (most recent call last):
    File "./run-tests.py", line 724, in run
      self.tearDown()
    File "./run-tests.py", line 805, in tearDown
      killdaemons(entry)
    File "./run-tests.py", line 540, in killdaemons
      logfn=vlog)
    File "...\tests\killdaemons.py", line 94, in killdaemons
      os.unlink(pidfile)
  WindowsError: [Error 32] The process cannot access the file because it is
     being used by another process: '...\\hgtests.zmpqj3\\child80\\daemon.pids'

Adrian suggested using util.posixfile, which works.  However, the 'mercurial'
package isn't in sys.path when invoking run-tests.py, and it isn't clear that
hacking[1] it in is a good thing (especially for test-run-tests.t, which uses an
installation in a temp folder).

I tried using ProcessMonitor to figure out what the other process is, but that
monitoring slows things down to such a degree that the issue doesn't occur.  I
was ready to blame the virus scanner, but it happens without that too.

Looking at the code, I don't see anything that would have the pid file open.
But I was able to get through about 20 full test runs without an issue with this
minor change, whereas before it was pretty certain to hit this at least once in
two or three runs.

[1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-May/097907.html
Adrian Buehlmann - June 5, 2017, 6:34 a.m.
On 2017-06-05 05:38, Matt Harbison wrote:
> # HG changeset patch
> # User Matt Harbison <matt_harbison@yahoo.com>
> # Date 1495503902 14400
> #      Mon May 22 21:45:02 2017 -0400
> # Node ID eaed97a21942af11c122607bf37f7399c68cae9d
> # Parent  ecc27f3123ea173f2dc66e20abbedad5741ea5e1
> killdaemons: close pid file before killing processes
> 
> With #serve enabled on Windows, I was getting occasional stacktraces like this:
> 
>   Errored test-hgweb-json.t: Traceback (most recent call last):
>     File "./run-tests.py", line 724, in run
>       self.tearDown()
>     File "./run-tests.py", line 805, in tearDown
>       killdaemons(entry)
>     File "./run-tests.py", line 540, in killdaemons
>       logfn=vlog)
>     File "...\tests\killdaemons.py", line 94, in killdaemons
>       os.unlink(pidfile)
>   WindowsError: [Error 32] The process cannot access the file because it is
>      being used by another process: '...\\hgtests.zmpqj3\\child80\\daemon.pids'

..

> Adrian suggested using util.posixfile, (..)

Not really. I was just trying to point out that if you use Python's
open(), you get the "used by another process" WindowsError on
os.unlink(), if the file in question is still open.

https://www.mercurial-scm.org/wiki/UnlinkingFilesOnWindows

> However, the 'mercurial'
> package isn't in sys.path when invoking run-tests.py, and it isn't clear that
> hacking[1] it in is a good thing (especially for test-run-tests.t, which uses an
> installation in a temp folder).
> 
> I tried using ProcessMonitor to figure out what the other process is, but that
> monitoring slows things down to such a degree that the issue doesn't occur.  I
> was ready to blame the virus scanner, but it happens without that too.
> 
> Looking at the code, I don't see anything that would have the pid file open.
> But I was able to get through about 20 full test runs without an issue with this
> minor change, whereas before it was pretty certain to hit this at least once in
> two or three runs.
> 
> [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2017-May/097907.html
> 
> diff --git a/tests/killdaemons.py b/tests/killdaemons.py
> --- a/tests/killdaemons.py
> +++ b/tests/killdaemons.py
> @@ -78,18 +78,20 @@
>          logfn = lambda s: s
>      # Kill off any leftover daemon processes
>      try:
> -        fp = open(pidfile)
> -        for line in fp:
> -            try:
> -                pid = int(line)
> -                if pid <= 0:
> -                    raise ValueError
> -            except ValueError:
> -                logfn('# Not killing daemon process %s - invalid pid'
> -                      % line.rstrip())
> -                continue
> +        pids = []
> +        with open(pidfile) as fp:
> +            for line in fp:
> +                try:
> +                    pid = int(line)
> +                    if pid <= 0:
> +                        raise ValueError
> +                except ValueError:
> +                    logfn('# Not killing daemon process %s - invalid pid'
> +                          % line.rstrip())
> +                    continue
> +                pids.append(pid)
> +        for pid in pids:
>              kill(pid, logfn, tryhard)
> -        fp.close()
>          if remove:
>              os.unlink(pidfile)
>      except IOError:

This looks good to me and the change is an improvement.

Making sure that files are closed before acting is always good for Windows.

I'd suggest taking this patch, but I haven't tested it.

Patch

diff --git a/tests/killdaemons.py b/tests/killdaemons.py
--- a/tests/killdaemons.py
+++ b/tests/killdaemons.py
@@ -78,18 +78,20 @@ 
         logfn = lambda s: s
     # Kill off any leftover daemon processes
     try:
-        fp = open(pidfile)
-        for line in fp:
-            try:
-                pid = int(line)
-                if pid <= 0:
-                    raise ValueError
-            except ValueError:
-                logfn('# Not killing daemon process %s - invalid pid'
-                      % line.rstrip())
-                continue
+        pids = []
+        with open(pidfile) as fp:
+            for line in fp:
+                try:
+                    pid = int(line)
+                    if pid <= 0:
+                        raise ValueError
+                except ValueError:
+                    logfn('# Not killing daemon process %s - invalid pid'
+                          % line.rstrip())
+                    continue
+                pids.append(pid)
+        for pid in pids:
             kill(pid, logfn, tryhard)
-        fp.close()
         if remove:
             os.unlink(pidfile)
     except IOError: