Patchwork D9019: procutil: avoid using os.fork() to implement runbgcommand

login
register
mail settings
Submitter phabricator
Date Sept. 14, 2020, 3:07 p.m.
Message ID <differential-rev-PHID-DREV-jp26pnxesc3tlbhchcm6-req@mercurial-scm.org>
Download mbox | patch
Permalink /patch/47167/
State New
Headers show

Comments

phabricator - Sept. 14, 2020, 3:07 p.m.
valentin.gatienbaron created this revision.
Herald added a reviewer: hg-reviewers.
Herald added a subscriber: mercurial-patches.

REVISION SUMMARY
  NB: this only works with py3. I'm hoping that maybe it won't be
  necessary to support py2 in a couple of months?
  
  We ran into the following deadlock:
  
  - some command creates an ssh peer, then raises without explicitly closing the peer (hg id + extension in our case)
  
  - dispatch catches the exception, calls ui.log('commandfinish', ..) (the sshpeer is still not closed), which calls logtoprocess, which calls procutil.runbgcommand.
  
  - in the child of runbgcommand's fork(), between the fork and the exec, the opening of file descriptors triggers a gc which runs the destructor for sshpeer, which waits on ssh's stderr being closed, which never happens since ssh's stderr is held open by the parent of the fork where said destructor hasn't run
  
  I don't know if there's more subtlety to it, because even though the
  problem is determistic, it is very fragile, so I didn't manage to
  reduce it.
  
  I can imagine three ways of tackling this problem:
  
  1. don't run any python between fork and exec in runbgcommand
  2. make the finalizer harmless after the fork
  3. close the peer without relying on gc behavior
  
  3 wouldn't help with calls to logtoprocess in the middle of commands.
  So this commit implements 1, and the next commit will do 2 for
  robustness (it might help with the forks from worker.py in hg pull
  -u).
  
  Performance wise: at low memory usage, it's an improvement. At higher
  memory usage, it's about 2x faster than before when ensurestart=True,
  but 2x slower when ensurestart=False. Not sure if that matters. The
  reason for that last bit is that the subprocess.Popen always waits for
  the execve to finish, and at high memory usage, execve is slow because
  it deallocates the large page table. Numbers and script:
  
    				 before after
        mem=1.0GB, ensurestart=True  52.1ms 26.0ms
        mem=1.0GB, ensurestart=False 14.7ms 26.0ms
        mem=0.5GB, ensurestart=True  23.2ms 11.2ms
        mem=0.5GB, ensurestart=False  6.2ms 11.3ms
        mem=0.2GB, ensurestart=True  15.7ms 7.4ms
        mem=0.2GB, ensurestart=False  4.3ms 8.1ms
        mem=0.0GB, ensurestart=True   2.3ms 0.7ms
        mem=0.0GB, ensurestart=False  0.8ms 0.8ms
    
        import time
        for memsize in [1_000_000_000, 500_000_000, 250_000_000, 0]:
            mem = 'a' * memsize
            for ensurestart in [True, False]:
                now = time.time()
                n = 100
                for i in range(n):
                    procutil.runbgcommand([b'true'], {}, ensurestart=ensurestart)
                after = time.time()
                ms = (after - now) / float(n) * 1000
                print(f'mem={memsize / 1e9:.1f}GB, ensurestart={ensurestart} -> {ms:.1f}ms')

REPOSITORY
  rHG Mercurial

BRANCH
  default

REVISION DETAIL
  https://phab.mercurial-scm.org/D9019

AFFECTED FILES
  mercurial/utils/procutil.py

CHANGE DETAILS




To: valentin.gatienbaron, #hg-reviewers
Cc: mercurial-patches, mercurial-devel

Patch

diff --git a/mercurial/utils/procutil.py b/mercurial/utils/procutil.py
--- a/mercurial/utils/procutil.py
+++ b/mercurial/utils/procutil.py
@@ -676,83 +676,53 @@ 
             # calls gui() while we're calling fork().
             gui()
 
-        # double-fork to completely detach from the parent process
-        # based on http://code.activestate.com/recipes/278731
+        if shell:
+            script = cmd
+        else:
+            if isinstance(cmd, bytes):
+                cmd = [cmd]
+            script = b' '.join(shellquote(x) for x in cmd)
         if record_wait is None:
-            pid = os.fork()
-            if pid:
-                if not ensurestart:
-                    # Even though we're not waiting on the child process,
-                    # we still must call waitpid() on it at some point so
-                    # it's not a zombie/defunct. This is especially relevant for
-                    # chg since the parent process won't die anytime soon.
-                    # We use a thread to make the overhead tiny.
-                    def _do_wait():
-                        os.waitpid(pid, 0)
+            # double-fork to completely detach from the parent process
+            script = b'( %s ) &' % script
+            start_new_session = True
+        else:
+            start_new_session = False
+            ensurestart = True
 
-                    t = threading.Thread(target=_do_wait)
-                    t.daemon = True
-                    t.start()
-                    return
-                # Parent process
-                (_pid, status) = os.waitpid(pid, 0)
-                if os.WIFEXITED(status):
-                    returncode = os.WEXITSTATUS(status)
-                else:
-                    returncode = -(os.WTERMSIG(status))
-                if returncode != 0:
-                    # The child process's return code is 0 on success, an errno
-                    # value on failure, or 255 if we don't have a valid errno
-                    # value.
-                    #
-                    # (It would be slightly nicer to return the full exception info
-                    # over a pipe as the subprocess module does.  For now it
-                    # doesn't seem worth adding that complexity here, though.)
-                    if returncode == 255:
-                        returncode = errno.EINVAL
-                    raise OSError(
-                        returncode,
-                        b'error running %r: %s'
-                        % (cmd, os.strerror(returncode)),
-                    )
-                return
-
-        returncode = 255
         try:
-            if record_wait is None:
-                # Start a new session
-                os.setsid()
-
-            stdin = open(os.devnull, b'r')
+            stdin = subprocess.DEVNULL
             if stdout is None:
-                stdout = open(os.devnull, b'w')
+                stdout = subprocess.DEVNULL
             if stderr is None:
-                stderr = open(os.devnull, b'w')
+                stderr = subprocess.DEVNULL
 
             # connect stdin to devnull to make sure the subprocess can't
             # muck up that stream for mercurial.
             p = subprocess.Popen(
-                cmd,
-                shell=shell,
+                script,
+                shell=True,
                 env=env,
                 close_fds=True,
                 stdin=stdin,
                 stdout=stdout,
                 stderr=stderr,
+                start_new_session=start_new_session,
             )
+        except Exception:
             if record_wait is not None:
-                record_wait(p.wait)
-            returncode = 0
-        except EnvironmentError as ex:
-            returncode = ex.errno & 0xFF
-            if returncode == 0:
-                # This shouldn't happen, but just in case make sure the
-                # return code is never 0 here.
-                returncode = 255
-        except Exception:
-            returncode = 255
-        finally:
-            # mission accomplished, this child needs to exit and not
-            # continue the hg process here.
-            if record_wait is None:
-                os._exit(returncode)
+                record_wait(255)
+            raise
+        if not ensurestart:
+            # Even though we're not waiting on the child process,
+            # we still must call waitpid() on it at some point so
+            # it's not a zombie/defunct. This is especially relevant for
+            # chg since the parent process won't die anytime soon.
+            # We use a thread to make the overhead tiny.
+            t = threading.Thread(target=lambda: p.wait)
+            t.daemon = True
+            t.start()
+        else:
+            returncode = p.wait
+            if record_wait is not None:
+                record_wait(returncode)