Patchwork D8339: dispatch: force \n for newlines on sys.std* streams (BC)

mail settings
Submitter phabricator
Date March 29, 2020, 8:31 p.m.
Message ID <>
Download mbox | patch
Permalink /patch/45939/
State Superseded
Headers show


phabricator - March 29, 2020, 8:31 p.m.
indygreg created this revision.
Herald added a subscriber: mercurial-devel.
Herald added a reviewer: hg-reviewers.

  The sys.std* streams behave differently on Python 3. On Python 3,
  these streams are an io.TextIOWrapper that wraps a binary buffer
  stored on a .buffer attribute. These TextIOWrapper instances
  normalize \n to os.linesep by default. On Windows, this means
  that \n is normalized to \r\n. So functions like print() which
  have an implicit end='\n' will actually emit \r\n for line endings.
  While most parts of Mercurial go through the ui.write() layer to
  print output, some code - notably in extensions and hooks - can use
  print(). If this code was using print() or otherwise writing to
  sys.std* on Windows, Mercurial would emit \r\n.
  In reality, pretty much everything on Windows reacts to \n just fine.
  Mercurial itself doesn't emit \r\n when going through the ui layer.
  Changing the sys.std* streams to not normalize line endings sounds
  like a scary change. But I think it is safe. It also makes Mercurial
  on Python 3 behave similarly to Python 2, which did not perform \r\n
  normalization in print() by default.
  .. bc:: sys.{stdout, stderr, stdin} now use \n line endings on Python 3

  rHG Mercurial





To: indygreg, #hg-reviewers
Cc: mercurial-devel
phabricator - March 29, 2020, 8:35 p.m.
indygreg added a comment.

  As scary as this patch sounds, I'm pretty sure it is safe, as I believe it restores compatibility with Python 2. Changing `sys.std*` to be binary streams instead of text streams would be a bigger BC break. And that is not a change I want to make, as this would invalidate assumptions in 3rd party code about the behavior of these streams on Python 3!

  rHG Mercurial



To: indygreg, #hg-reviewers
Cc: mercurial-devel


diff --git a/mercurial/ b/mercurial/
--- a/mercurial/
+++ b/mercurial/
@@ -10,6 +10,7 @@ 
 import difflib
 import errno
 import getopt
+import io
 import os
 import pdb
 import re
@@ -144,7 +145,50 @@ 
 if pycompat.ispy3:
     def initstdio():
-        pass
+        # stdio streams on Python 3 are io.TextIOWrapper instances proxying another
+        # buffer. These streams will normalize \n to \r\n by default. Mercurial's
+        # preferred mechanism for writing output (ui.write()) uses io.BufferedWriter
+        # instances, which write to the underlying stdio file descriptor in binary
+        # mode. ui.write() uses \n for line endings and no line ending normalization
+        # is attempted through this interface. This "just works," even if the system
+        # preferred line ending is not \n.
+        #
+        # But some parts of Mercurial (e.g. hooks) can still send data to sys.stdout
+        # and sys.stderr. They will inherit the line ending normalization settings,
+        # potentially causing e.g. \r\n to be emitted. Since emitting \n should
+        # "just work," here we change the sys.* streams to disable line ending
+        # normalization, ensuring compatibility with our ui type.
+        # write_through is new in Python 3.7.
+        kwargs = {
+            "newline": "\n",
+            "line_buffering": sys.stdout.line_buffering,
+        }
+        if util.safehasattr(sys.stdout, "write_through"):
+            kwargs["write_through"] = sys.stdout.write_through
+        sys.stdout = io.TextIOWrapper(
+            sys.stdout.buffer, sys.stdout.encoding, sys.stdout.errors, **kwargs
+        )
+        kwargs = {
+            "newline": "\n",
+            "line_buffering": sys.stderr.line_buffering,
+        }
+        if util.safehasattr(sys.stderr, "write_through"):
+            kwargs["write_through"] = sys.stderr.write_through
+        sys.stderr = io.TextIOWrapper(
+            sys.stderr.buffer, sys.stderr.encoding, sys.stderr.errors, **kwargs
+        )
+        # No write_through on read-only stream.
+        sys.stdin = io.TextIOWrapper(
+            sys.stdin.buffer,
+            sys.stdin.encoding,
+            sys.stdin.errors,
+            # None is universal newlines mode.
+            newline=None,
+            line_buffering=sys.stdin.line_buffering,
+        )
     def _silencestdio():
         for fp in (sys.stdout, sys.stderr):