Python Urllib2 SSL error
Question: Python 2.7.9 is now much more strict about SSL certificate verification.
Awesome!

I'm not surprised that programs that were working before are now getting
CERTIFICATE_VERIFY_FAILED errors. But I can't seem to get them working
(without disabling certificate verification entirely).

One program was using urllib2 to connect to Amazon S3 over https.

I download the root CA certificate into a file called "verisign.pem" and try
this:

    
    
    import urllib2, ssl
    context = ssl.create_default_context()
    context.load_verify_locations(cafile = "./verisign.pem")
    print context.get_ca_certs()
    urllib2.urlopen("https://bucket.s3.amazonaws.com/", context=context)
    

and I still get CERTIFICATE_VERIFY_FAILED errors, even though the root CA is
printed out correctly in line 4.

openssl can connect to this server fine. In fact, here is the command I used
to get the CA cert:

    
    
    openssl s_client -showcerts -connect bucket.s3.amazonaws.com:443 < /dev/null
    

I took the last cert in the chain and put it in a PEM file, which openssl
reads fine. It's a Verisign certificate with:

    
    
    Serial number: 35:97:31:87:f3:87:3a:07:32:7e:ce:58:0c:9b:7e:da
    Subject key identifier: 7F:D3:65:A7:C2:DD:EC:BB:F0:30:09:F3:43:39:FA:02:AF:33:31:33
    SHA1 fingerprint: F4:A8:0A:0C:D1:E6:CF:19:0B:8C:BC:6F:BC:99:17:11:D4:82:C9:D0
    

Any ideas how to get this working with validation enabled?


Answer: To summarize the comments about the cause of the problem and explain the real
problem in more detail:

If you check the trust chain for the OpenSSL client you get the following:

    
    
     [0] 54:7D:B3:AC:BF:... /CN=*.s3.amazonaws.com 
     [1] 5D:EB:8F:33:9E:... /CN=VeriSign Class 3 Secure Server CA - G3
     [2] F4:A8:0A:0C:D1:... /CN=VeriSign Class 3 Public Primary Certification Authority - G5
    [OT] A1:DB:63:93:91:... /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
    

The first certificate [0] is the leaf certificate sent by the server. The
following certifcates [1] and [2] are chain certificates sent by the server.
The last certificate [OT] is the trusted root certificate, which is not sent
by the server but is in the local storage of trusted CA. Each certificate in
the chain is signed by the next one and the last certificate [OT] is trusted,
so the trust chain is complete.

If you check the trust chain instead by a browser (e.g. Google Chrome using
the NSS library) you get the following chain:

    
    
     [0] 54:7D:B3:AC:BF:... /CN=*.s3.amazonaws.com 
     [1] 5D:EB:8F:33:9E:... /CN=VeriSign Class 3 Secure Server CA - G3
    [NT] 4E:B6:D5:78:49:... /CN=VeriSign Class 3 Public Primary Certification Authority - G5
    

Here [0] and [1] are again sent by the server, but [NT] is the trusted root
certificate. While this looks from the subject exactly like the chain
certificate [2] the fingerprint says that the certificates are different. If
you would take a closer looks at the certificates [2] and [NT] you would see,
that the public key inside the certificate is the same and thus both [2] and
[NT] can be used to verify the signature for [1] and thus can be used to build
the trust chain.

This means, that while the server sends the same certificate chain in all
cases there are multiple ways to verify the chain up to a trusted root
certificate. How this is done depends on the SSL library and on the known
trusted root certificates:

    
    
                              [0] (*.s3.amazonaws.com)
                               |
                              [1] (Verisign G3) --------------------------\
                               |                                          |
          /------------------ [2] (Verisign G5 F4:A8:0A:0C:D1...)         |
          |                                                               |
          |              certificates sent by server                      |
     .....|...............................................................|................
          |              locally trusted root certificates                |
          |                                                               |
         [OT] Public Primary Certification Authority        [NT] Verisign G5 4E:B6:D5:78:49
         OpenSSL library                                    Google Chrome (NSS library)
    

But the question remains, why your verification was unsuccessful. What you did
was to take the trusted root certificate used by the browser (Verisign G5
4E:B6:D5:78:49) together with OpenSSL. But the verification in browser (NSS)
and OpenSSL work slightly different:

  * NSS: build trust chain from certificates send by the server. Stop building the chain when we got a certificate signed by any of the locally trusted root certificates.
  * OpenSSL_ build trust chain from the certificates sent by the server. After this is done check if we have a trusted root certificate signing the latest certificate in the chain.

Because of this subtle difference OpenSSL is not able to verify the chain
[0],[1],[2] against root certificate [NT], because this certificate does not
sign the latest element in chain [2] but instead [1]. If the server would
instead only sent a chain of [0],[1] then the verification would succeed.

This is a [long known
bug](http://rt.openssl.org/Ticket/Display.html?id=2732&user=guest&pass=guest)
and there exist
[patches](https://www.google.de/?q=patch%20x509_v_flag_trusted_first) and
hopefully the issue if finally addressed in OpenSSL 1.0.2 with the
introduction of the `X509_V_FLAG_TRUSTED_FIRST` option.

