Patchwork [v2] ssl: only use the dummy cert hack if using an Apple Python (issue4410)

login
register
mail settings
Submitter Mads Kiilerich
Date Oct. 17, 2014, 4:57 p.m.
Message ID <4495fd290ef36d4cd65f.1413565043@ssl.google-analytics.com>
Download mbox | patch
Permalink /patch/6371/
State Accepted
Headers show

Comments

Mads Kiilerich - Oct. 17, 2014, 4:57 p.m.
# HG changeset patch
# User Mads Kiilerich <madski@unity3d.com>
# Date 1413564972 -7200
#      Fri Oct 17 18:56:12 2014 +0200
# Node ID 4495fd290ef36d4cd65f4b613e9b3da7f87aff15
# Parent  840be5ca03e1db16ba994e55597771c418166c97
ssl: only use the dummy cert hack if using an Apple Python (issue4410)

The hack for using certificate store in addition to the provided CAs resides in
Apple's OpenSSL. Apple's own Pythons will use it, but other custom built
Pythons might use a custom built OpenSSL without that hack and will fail when
exposed to the dummy cacert introduced in d7f7f1860f00.

There do not seem to be a simple way to check from Python if we are using a
patched OpenSSL or if it is an Apple OpenSSL.

Instead, check if the Python executable resides in /usr/bin/python* or in
/System/Library/Frameworks/Python.framework/ and assume that all Pythons found
there will be native Pythons using the patched OpenSSL.

Custom built Pythons will not get the benefit of using the CAs from the
certificate store.
Gregory Szorc - Oct. 18, 2014, 1:46 a.m.
On 10/17/14 9:57 AM, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich <madski@unity3d.com>
> # Date 1413564972 -7200
> #      Fri Oct 17 18:56:12 2014 +0200
> # Node ID 4495fd290ef36d4cd65f4b613e9b3da7f87aff15
> # Parent  840be5ca03e1db16ba994e55597771c418166c97
> ssl: only use the dummy cert hack if using an Apple Python (issue4410)
>
> The hack for using certificate store in addition to the provided CAs resides in
> Apple's OpenSSL. Apple's own Pythons will use it, but other custom built
> Pythons might use a custom built OpenSSL without that hack and will fail when
> exposed to the dummy cacert introduced in d7f7f1860f00.
>
> There do not seem to be a simple way to check from Python if we are using a
> patched OpenSSL or if it is an Apple OpenSSL.
>
> Instead, check if the Python executable resides in /usr/bin/python* or in
> /System/Library/Frameworks/Python.framework/ and assume that all Pythons found
> there will be native Pythons using the patched OpenSSL.
>
> Custom built Pythons will not get the benefit of using the CAs from the
> certificate store.

Looking into this some more, the ssl package in Python 2.7.9 (which 
isn't even released yet!) has an SSLContext class 
(https://docs.python.org/2/library/ssl.html#ssl-contexts) that allows 
you to read CA info. We could conceivably use this (if available) to 
determine whether the CA store is empty and whether applying the dummy 
cert hack is necessary. http://bugs.python.org/issue21308 tracked this 
backport. It's quite possible they also fixed CA discovery on OS X - I 
haven't looked into the details too much. But we still have to support 
the masses that won't be on 2.7.9, and this includes not having the ssl 
package on <2.6.

There's probably room for a follow-up to make `hg debuginstall` check 
the state of the system CA certs by performing a request to a well-known 
hostname like www.mozilla.org and recommending actions if certificate 
verification fails. Do the APIs in Python 2.4 even expose these kinds of 
failure details? IIRC they don't: 2.4 did TLS in a non-visible manner in 
C land.

Speaking of 2.4, weren't we going to drop support for it and 2.5...
Gregory Szorc - Oct. 18, 2014, 2:08 a.m.
> On Oct 17, 2014, at 18:46, Gregory Szorc <gregory.szorc@gmail.com> wrote:
> 
>> On 10/17/14 9:57 AM, Mads Kiilerich wrote:
>> # HG changeset patch
>> # User Mads Kiilerich <madski@unity3d.com>
>> # Date 1413564972 -7200
>> #      Fri Oct 17 18:56:12 2014 +0200
>> # Node ID 4495fd290ef36d4cd65f4b613e9b3da7f87aff15
>> # Parent  840be5ca03e1db16ba994e55597771c418166c97
>> ssl: only use the dummy cert hack if using an Apple Python (issue4410)
>> 
>> The hack for using certificate store in addition to the provided CAs resides in
>> Apple's OpenSSL. Apple's own Pythons will use it, but other custom built
>> Pythons might use a custom built OpenSSL without that hack and will fail when
>> exposed to the dummy cacert introduced in d7f7f1860f00.
>> 
>> There do not seem to be a simple way to check from Python if we are using a
>> patched OpenSSL or if it is an Apple OpenSSL.
>> 
>> Instead, check if the Python executable resides in /usr/bin/python* or in
>> /System/Library/Frameworks/Python.framework/ and assume that all Pythons found
>> there will be native Pythons using the patched OpenSSL.
>> 
>> Custom built Pythons will not get the benefit of using the CAs from the
>> certificate store.
> 
> Looking into this some more, the ssl package in Python 2.7.9 (which isn't even released yet!) has an SSLContext class (https://docs.python.org/2/library/ssl.html#ssl-contexts) that allows you to read CA info. We could conceivably use this (if available) to determine whether the CA store is empty and whether applying the dummy cert hack is necessary. http://bugs.python.org/issue21308 tracked this backport. It's quite possible they also fixed CA discovery on OS X - I haven't looked into the details too much. But we still have to support the masses that won't be on 2.7.9, and this includes not having the ssl package on <2.6.
> 
> There's probably room for a follow-up to make `hg debuginstall` check the state of the system CA certs by performing a request to a well-known hostname like www.mozilla.org and recommending actions if certificate verification fails. Do the APIs in Python 2.4 even expose these kinds of failure details? IIRC they don't: 2.4 did TLS in a non-visible manner in C land.
> 

Ignore this last bit. I should not even try to use my brain when I take a day off.

> Speaking of 2.4, weren't we going to drop support for it and 2.5...
>
Mads Kiilerich - Oct. 18, 2014, 10:45 a.m.
On 10/18/2014 03:46 AM, Gregory Szorc wrote:
> Looking into this some more, the ssl package in Python 2.7.9 (which 
> isn't even released yet!) has an SSLContext class 
> (https://docs.python.org/2/library/ssl.html#ssl-contexts) that allows 
> you to read CA info. We could conceivably use this (if available) to 
> determine whether the CA store is empty and whether applying the dummy 
> cert hack is necessary. http://bugs.python.org/issue21308 tracked this 
> backport. It's quite possible they also fixed CA discovery on OS X - I 
> haven't looked into the details too much. But we still have to support 
> the masses that won't be on 2.7.9, and this includes not having the 
> ssl package on <2.6.

Yes, we should support the new "2.8" APIs too. That is not implemented 
yet. So far it is more important to support >= 2.6.

Python (and thus Mercurial) do not really have ssl support before 2.6.

> There's probably room for a follow-up to make `hg debuginstall` check 
> the state of the system CA certs by performing a request to a 
> well-known hostname like www.mozilla.org and recommending actions if 
> certificate verification fails.

That might be a good idea.

Some might dislike that it connects to 3rd party sites without asking 
for permission, so it should perhaps not be the default.

Also, if we could give a good advice, we could just as well automate it 
and ship the curl/mozilla ca cert list as fall back. That would take 
most users from "insecure, but not our fault" to "mostly secure, but our 
fault when not really secure anyway".

(Btw: it would be very nice if Mozilla could provide their trusted CA 
certs directly as PEMs so people didn't have to visit haxx and decide if 
they trust them.)

> Speaking of 2.4, weren't we going to drop support for it and 2.5...

So far that is wishful thinking.

One of the major reasons to keep it supported has been weakened by 'make 
docker-centos5' creating EL5 rpms that contains their own Python 2.7.x.

/Mads
Matt Mackall - Oct. 18, 2014, 8:22 p.m.
On Fri, 2014-10-17 at 18:57 +0200, Mads Kiilerich wrote:
> # HG changeset patch
> # User Mads Kiilerich <madski@unity3d.com>
> # Date 1413564972 -7200
> #      Fri Oct 17 18:56:12 2014 +0200
> # Node ID 4495fd290ef36d4cd65f4b613e9b3da7f87aff15
> # Parent  840be5ca03e1db16ba994e55597771c418166c97
> ssl: only use the dummy cert hack if using an Apple Python (issue4410)

Queued for default, thanks.
Gregory Szorc - Oct. 20, 2014, 2:58 a.m.
On 10/18/14 3:45 AM, Mads Kiilerich wrote:
> (Btw: it would be very nice if Mozilla could provide their trusted CA
> certs directly as PEMs so people didn't have to visit haxx and decide if
> they trust them.)

I agree. I did a bit more research and posted to the mailing list that 
does CA foo and asked why Mozilla can't host a PEM file. With any luck 
this situation will soon change.

https://groups.google.com/d/msg/mozilla.dev.security.policy/FYIBEF_AVMI/D_vaVSaruXUJ

Patch

diff --git a/mercurial/sslutil.py b/mercurial/sslutil.py
--- a/mercurial/sslutil.py
+++ b/mercurial/sslutil.py
@@ -88,6 +88,20 @@  def _verifycert(cert, hostname):
 # We COMPLETELY ignore CERT_REQUIRED on Python <= 2.5, as it's totally
 # busted on those versions.
 
+def _plainapplepython():
+    """return true if this seems to be a pure Apple Python that
+    * is unfrozen and presumably has the whole mercurial module in the file
+      system
+    * presumably is an Apple Python that uses Apple OpenSSL which has patches
+      for using system certificate store CAs in addition to the provided
+      cacerts file
+    """
+    if sys.platform != 'darwin' or util.mainfrozen():
+        return False
+    exe = (sys.executable or '').lower()
+    return (exe.startswith('/usr/bin/python') or
+            exe.startswith('/system/library/frameworks/python.framework/'))
+
 def sslkwargs(ui, host):
     forcetls = ui.configbool('ui', 'tls', default=True)
     if forcetls:
@@ -104,7 +118,7 @@  def sslkwargs(ui, host):
         cacerts = util.expandpath(cacerts)
         if not os.path.exists(cacerts):
             raise util.Abort(_('could not find web.cacerts: %s') % cacerts)
-    elif cacerts is None and sys.platform == 'darwin' and not util.mainfrozen():
+    elif cacerts is None and _plainapplepython():
         dummycert = os.path.join(os.path.dirname(__file__), 'dummycert.pem')
         if os.path.exists(dummycert):
             ui.debug('using %s to enable OS X system CA\n' % dummycert)
diff --git a/tests/test-https.t b/tests/test-https.t
--- a/tests/test-https.t
+++ b/tests/test-https.t
@@ -115,7 +115,8 @@  Test server address cannot be reused
 #endif
   $ cd ..
 
-OS X has a dummy CA cert that enables use of the system CA store
+OS X has a dummy CA cert that enables use of the system CA store when using
+Apple's OpenSSL. This trick do not work with plain OpenSSL.
 
   $ DISABLEOSXDUMMYCERT=
 #if osx