Patchwork [4,of,5] py3: stop subscripting socket.error

login
register
mail settings
Submitter Matt Harbison
Date Dec. 16, 2018, 5:36 a.m.
Message ID <op.zt3ojjt39lwrgf@envy>
Download mbox | patch
Permalink /patch/37201/
State New
Headers show

Comments

Matt Harbison - Dec. 16, 2018, 5:36 a.m.
On Sat, 15 Dec 2018 23:58:13 -0500, Matt Harbison <mharbison72@gmail.com>  
wrote:

> On Mon, 10 Dec 2018 07:12:41 -0500, Yuya Nishihara <yuya@tcha.org> wrote:
>
>> On Sun, 09 Dec 2018 23:39:33 -0500, Matt Harbison wrote:
>>> I ended up with similar stdio errors trying to track down the bad http
>>> status failures:
>>>
>>> Traceback (most recent call last):
>>>    File "c:\Users\Matt\hg\mercurial\ui.py", line 1033, in _writenobuf
>>>      dest.flush()
>>> OSError: [WinError 1] Incorrect function
>>>
>>> During handling of the above exception, another exception occurred:
>>>
>>> Traceback (most recent call last):
>>>    File "c:\Users\Matt\hg\mercurial\hgweb\server.py", line 192, in  
>>> do_hgweb
>>>      for chunk in self.server.application(env, self._start_response):
>>>    File "c:\Users\Matt\hg\mercurial\hgweb\hgweb_mod.py", line 310, in
>>> run_wsgi
>>>      for r in self._runwsgi(req, res, repo):
>>>    File "c:\Users\Matt\hg\mercurial\hgweb\hgweb_mod.py", line 430, in
>>> _runwsgi
>>>      return getattr(webcommands, cmd)(rctx)
>>>    File "c:\Users\Matt\hg\mercurial\hgweb\webcommands.py", line 861, in
>>> comparison
>>>      context = parsecontext(web.config('web', 'comparisoncontext',  
>>> '5'))
>>>    File "c:\Users\Matt\hg\mercurial\hgweb\hgweb_mod.py", line 117, in  
>>> config
>>>      untrusted=untrusted)
>>>    File "c:\Users\Matt\hg\mercurial\ui.py", line 517, in config
>>>      untrusted=untrusted)
>>>    File "c:\Users\Matt\hg\mercurial\ui.py", line 536, in _config
>>>      self.develwarn(msg, 2, 'warn-config-unknown')
>>>    File "c:\Users\Matt\hg\mercurial\ui.py", line 1790, in develwarn
>>>      % (msg, fname, lineno, fmsg))
>>>    File "c:\Users\Matt\hg\mercurial\ui.py", line 996, in write_err
>>>      self._write(self._ferr, *args, **opts)
>>>    File "c:\Users\Matt\hg\mercurial\ui.py", line 1006, in _write
>>>      self._writenobuf(dest, *args, **opts)
>>>    File "c:\Users\Matt\hg\mercurial\ui.py", line 1039, in _writenobuf
>>>      raise error.StdioError(err)
>>> mercurial.error.StdioError: [Errno 22] Incorrect function
>>>
>>> I see that there's a TODO for pycompat.{stdin,stdout,stderr} to use a
>>> silly wrapper instead of .buffer.  But this feeble attempt didn't seem  
>>> to
>>> help:
>>
>> That's unrelated issue. A pseudo file object might not have .buffer,  
>> but a
>> real stdio object should have one.
>>
>> Perhaps, the underlying file descriptor would be messed up in Windows  
>> way.
>> If we can know that prior to calling write(), stdio fds can be  
>> reattached to
>> NUL. But I don't know if that's possible.
>
> I'm not sure what to do with this info yet, but I just noticed that  
> pager is also messed up on py3- the debug message about spawning the  
> pager prints, but no output for the diff.  Use --pager=no, and it shows  
> up.  So maybe there's a general problem with stdio of spawned children.

To be specific, this:



Results in this:

$ py -3 ../hg diff -r .^^ --debug
*** failed to import extension evolve: No module named 'evolve'
obsolete feature not enabled but 177518 markers found!
starting pager for command 'diff'

Traceback (most recent call last):
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1024, in _writenobuf
     color.win32print(self, dest.write, msg, **opts)
   File "c:\Users\Matt\hg\mercurial\color.py", line 528, in win32print
     writefunc(m.group(2))
   File "c:\Users\Matt\hg\mercurial\windows.py", line 202, in write
     self.fp.write(s[start:end])
OSError: [WinError 1] Incorrect function

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1211, in _runpager
     self.write('test output\n')
   File "c:\Users\Matt\hg\mercurial\ui.py", line 993, in write
     self._write(self._fout, *args, **opts)
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1006, in _write
     self._writenobuf(dest, *args, **opts)
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1039, in _writenobuf
     raise error.StdioError(err)
mercurial.error.StdioError: [Errno 22] Incorrect function
abort: Incorrect function


Absolutely no problem running this code under py2.  Changing self.write()  
to self.error() results in the text making it to the screen, followed by  
an exception:

test output
Traceback (most recent call last):
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1061, in flush
     self._ferr.flush()
OSError: [WinError 1] Incorrect function

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1024, in _writenobuf
     color.win32print(self, dest.write, msg, **opts)
   File "c:\Users\Matt\hg\mercurial\color.py", line 532, in win32print
     ui.flush()
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1064, in flush
     raise error.StdioError(err)
mercurial.error.StdioError: [Errno 22] Incorrect function

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1211, in _runpager
     self.error('test output\n')
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1516, in error
     self._writemsg(self._fmsgerr, type='error', *msg, **opts)
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1045, in _writemsg
     _writemsgwith(self._write, dest, *args, **opts)
   File "c:\Users\Matt\hg\mercurial\ui.py", line 2048, in _writemsgwith
     write(dest, *args, **opts)
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1006, in _write
     self._writenobuf(dest, *args, **opts)
   File "c:\Users\Matt\hg\mercurial\ui.py", line 1039, in _writenobuf
     raise error.StdioError(err)
mercurial.error.StdioError: [Errno 22] Incorrect function
abort: Incorrect function
Yuya Nishihara - Dec. 16, 2018, 5:55 a.m.
On Sun, 16 Dec 2018 00:36:45 -0500, Matt Harbison wrote:
> > I'm not sure what to do with this info yet, but I just noticed that  
> > pager is also messed up on py3- the debug message about spawning the  
> > pager prints, but no output for the diff.  Use --pager=no, and it shows  
> > up.  So maybe there's a general problem with stdio of spawned children.
> 
> To be specific, this:
> 
> diff --git a/mercurial/ui.py b/mercurial/ui.py
> --- a/mercurial/ui.py
> +++ b/mercurial/ui.py
> @@ -1206,6 +1207,14 @@ class ui(object):
>               pager.stdin.close()
>               pager.wait()
> 
> +        try:
> +           self.write('test output\n')
> +           self.flush()
> +        except Exception:
> +            killpager()
> +            self.traceback(force=True)
> +            raise
> +
>           return True
> 
>       @property
> 
> 
> Results in this:
> 
> $ py -3 ../hg diff -r .^^ --debug
> *** failed to import extension evolve: No module named 'evolve'
> obsolete feature not enabled but 177518 markers found!
> starting pager for command 'diff'
> 
> Traceback (most recent call last):
>    File "c:\Users\Matt\hg\mercurial\ui.py", line 1024, in _writenobuf
>      color.win32print(self, dest.write, msg, **opts)
>    File "c:\Users\Matt\hg\mercurial\color.py", line 528, in win32print
>      writefunc(m.group(2))
>    File "c:\Users\Matt\hg\mercurial\windows.py", line 202, in write
>      self.fp.write(s[start:end])
> OSError: [WinError 1] Incorrect function

That might be because the stdio files are backed by the Windows-ish type.
Can you try PYTHONLEGACYWINDOWSSTDIO=1?

https://www.python.org/dev/peps/pep-0528/

Python 3 shifted away from being Unix scripting language in many ways.
Matt Harbison - Dec. 16, 2018, 6:19 a.m.
On Sun, 16 Dec 2018 00:55:20 -0500, Yuya Nishihara <yuya@tcha.org> wrote:

> On Sun, 16 Dec 2018 00:36:45 -0500, Matt Harbison wrote:
>> > I'm not sure what to do with this info yet, but I just noticed that
>> > pager is also messed up on py3- the debug message about spawning the
>> > pager prints, but no output for the diff.  Use --pager=no, and it  
>> shows
>> > up.  So maybe there's a general problem with stdio of spawned  
>> children.
>>
>> To be specific, this:
>>
>> diff --git a/mercurial/ui.py b/mercurial/ui.py
>> --- a/mercurial/ui.py
>> +++ b/mercurial/ui.py
>> @@ -1206,6 +1207,14 @@ class ui(object):
>>               pager.stdin.close()
>>               pager.wait()
>>
>> +        try:
>> +           self.write('test output\n')
>> +           self.flush()
>> +        except Exception:
>> +            killpager()
>> +            self.traceback(force=True)
>> +            raise
>> +
>>           return True
>>
>>       @property
>>
>>
>> Results in this:
>>
>> $ py -3 ../hg diff -r .^^ --debug
>> *** failed to import extension evolve: No module named 'evolve'
>> obsolete feature not enabled but 177518 markers found!
>> starting pager for command 'diff'
>>
>> Traceback (most recent call last):
>>    File "c:\Users\Matt\hg\mercurial\ui.py", line 1024, in _writenobuf
>>      color.win32print(self, dest.write, msg, **opts)
>>    File "c:\Users\Matt\hg\mercurial\color.py", line 528, in win32print
>>      writefunc(m.group(2))
>>    File "c:\Users\Matt\hg\mercurial\windows.py", line 202, in write
>>      self.fp.write(s[start:end])
>> OSError: [WinError 1] Incorrect function
>
> That might be because the stdio files are backed by the Windows-ish type.
> Can you try PYTHONLEGACYWINDOWSSTDIO=1?
>
> https://www.python.org/dev/peps/pep-0528/
>
> Python 3 shifted away from being Unix scripting language in many ways.

Yep, that fixed both pager and the strange server problems, thanks.  I  
need more sleep before digesting that PEP, but skimming it seems to imply  
that we need to explicitly encode to stdio on Windows?  How will that  
interact with the existing encoding functions?
Yuya Nishihara - Dec. 16, 2018, 6:27 a.m.
On Sun, 16 Dec 2018 01:19:47 -0500, Matt Harbison wrote:
> On Sun, 16 Dec 2018 00:55:20 -0500, Yuya Nishihara <yuya@tcha.org> wrote:
> 
> > On Sun, 16 Dec 2018 00:36:45 -0500, Matt Harbison wrote:
> >> > I'm not sure what to do with this info yet, but I just noticed that
> >> > pager is also messed up on py3- the debug message about spawning the
> >> > pager prints, but no output for the diff.  Use --pager=no, and it  
> >> shows
> >> > up.  So maybe there's a general problem with stdio of spawned  
> >> children.
> >>
> >> To be specific, this:
> >>
> >> diff --git a/mercurial/ui.py b/mercurial/ui.py
> >> --- a/mercurial/ui.py
> >> +++ b/mercurial/ui.py
> >> @@ -1206,6 +1207,14 @@ class ui(object):
> >>               pager.stdin.close()
> >>               pager.wait()
> >>
> >> +        try:
> >> +           self.write('test output\n')
> >> +           self.flush()
> >> +        except Exception:
> >> +            killpager()
> >> +            self.traceback(force=True)
> >> +            raise
> >> +
> >>           return True
> >>
> >>       @property
> >>
> >>
> >> Results in this:
> >>
> >> $ py -3 ../hg diff -r .^^ --debug
> >> *** failed to import extension evolve: No module named 'evolve'
> >> obsolete feature not enabled but 177518 markers found!
> >> starting pager for command 'diff'
> >>
> >> Traceback (most recent call last):
> >>    File "c:\Users\Matt\hg\mercurial\ui.py", line 1024, in _writenobuf
> >>      color.win32print(self, dest.write, msg, **opts)
> >>    File "c:\Users\Matt\hg\mercurial\color.py", line 528, in win32print
> >>      writefunc(m.group(2))
> >>    File "c:\Users\Matt\hg\mercurial\windows.py", line 202, in write
> >>      self.fp.write(s[start:end])
> >> OSError: [WinError 1] Incorrect function
> >
> > That might be because the stdio files are backed by the Windows-ish type.
> > Can you try PYTHONLEGACYWINDOWSSTDIO=1?
> >
> > https://www.python.org/dev/peps/pep-0528/
> >
> > Python 3 shifted away from being Unix scripting language in many ways.
> 
> Yep, that fixed both pager and the strange server problems, thanks.  I  
> need more sleep before digesting that PEP, but skimming it seems to imply  
> that we need to explicitly encode to stdio on Windows?  How will that  
> interact with the existing encoding functions?

I don't think we'll need the new Windows wrapper to get the Py2-level
Windows support. Instead, we'll have to make our exewrapper enable all
the "Legacy" options.

https://docs.python.org/3/c-api/init.html#global-configuration-variables

Patch

diff --git a/mercurial/ui.py b/mercurial/ui.py
--- a/mercurial/ui.py
+++ b/mercurial/ui.py
@@ -1206,6 +1207,14 @@  class ui(object):
              pager.stdin.close()
              pager.wait()

+        try:
+           self.write('test output\n')
+           self.flush()
+        except Exception:
+            killpager()
+            self.traceback(force=True)
+            raise
+
          return True

      @property