AppEngine urlfetch validate_certificate=False/None not being respected
Question: In the AppEngine developer appserver I am getting an error like this:

    
    
    SSLCertificateError: Invalid and/or missing SSL certificate for URL ...
    

when I am making a fetch like this to an `https` server with a self-signed
certificate (almost always `localhost` port-forwarded over ssh to a vm):

    
    
    result = urlfetch.fetch(url=url, method=method, payload=payload,
                            deadline=DEADLINE, validate_certificate=None)
    

One would not expect SSL failures for invalid certificates where
`validate_certificate` is `False`, though this is quite possibly a side-effect
of the 2.7.9 policy in Python to always validate ssl certificates.

Note that passing `False` (instead of `None`) for `validate_certificate` does
not work either.

This problem happens on Python 2.7.9-10 via Homebrew/XCode on OS X 10.10.2-4
with AppEngine 1.9.18 through 1.19.26.

There are issues (e.g.
[12096](https://code.google.com/p/googleappengine/issues/detail?id=12096))
about this on Google App Engine, but I am looking for a workaround.

Here's what I've tried to work around this:

  1. Add the certificate to the Mac's login keychain (works in the browser, not from Python)

  2. Add the certificate to `app-engine-python/lib/cacerts/cacerts.txt` and/or `./lib/cacerts/urlfetch_cacerts.txt` (though this probably requires turning verification _on_ for it to work, since that appears to be the only case where they are used) with e.g.

> $ echo >> /usr/local/share/app-engine-
> python/lib/cacerts/urlfetch_cacerts.txt
>
> $ openssl x509 -subject -in server.crt >> /usr/local/share/app-engine-
> python/lib/cacerts/urlfetch_cacerts.txt

  3. Disable ssl HTTPs checking with the [PEP-0476](https://www.python.org/dev/peps/pep-0476/) workaround i.e.

`ssl._create_default_https_context = ssl._create_unverified_context`

at or after `import ssl` (around line 1149) of
[`google/appengine/dist27/python_std_lib/httplib.py`](https://github.com/GoogleCloudPlatform/appengine-
python-vm-
runtime/blob/master/python_vm_runtime/google/appengine/dist27/python_std_lib/httplib.py#L1149)

This is particularly problematic on Mac since downgrading as of XCode 7/OS X
El Capital is no longer a practical option.

A preferable workaround would not involve monkey-patching the AppEngine code
proper every time the development appserver is updated.

* * *

**EDIT**

Note that the Mac builtin OpenSSL certificates are stored in
`/System/Library/OpenSSL`, which is protected with
[SIP/rootlessness](http://apple.stackexchange.com/a/193379/5522), which
frankly is a pain to muck with and a worthwhile feature to keep if we can.

I have verified that the certificate validates by using `openssl s_client
-connect localhost:7500 -CAfile server.pem`.

It's been added to the Keychain and to `/usr/local/etc/openssl/certs` with the
`hash.#` format where the hash comes from `openssl x509 -subject_hash -in
server.pem` (or the homebrew ssl, namely
`/usr/local/Cellar/openssl/1.0.2d_1/bin/openssl`). In which case
`/usr/local/Cellar/openssl/1.0.2d_1/bin/openssl s_client -connect
localhost:7500` verifies the certificate (but python still does not).

I have tried using the homebrew version of python and openssl, but to no
avail. Running the following in Python seems to always fail;

    
    
    ./pve/bin/python -c "import requests; requests.get('https://localhost:7500')"
    

This also fails where `SSL_CERT_FILE` is set to the server's certificate (i.e.
for added measure one might expect it to work since the `openssl` command
essentially works like this), and also fails where `SSL_CERT_PATH` is set to
`/usr/local/etc/openssl/certs`.

Note, `pve` is a virtual env where `help(ssl)` shows a `FILE` of
`/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py`

Further verifying that homebrew Python's `_ssl.so` links to homebrew's openssl
I ran:

xcrun otool -L
/usr/local/Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-
dynload/_ssl.so

which returns

>
> ./Cellar/python/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-
> dynload/_ssl.so:
>
> /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0,
> current version 1.0.0)
>
> /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version
> 1.0.0, current version 1.0.0)
>
> /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version
> 1225.1.1)

If one runs `brew info openssl` it remarks under `CAVEATS`:

> A CA file has been bootstrapped using certificates from the system keychain.
> To add additional certificates, place .pem files in
> /usr/local/etc/openssl/certs

but clearly for some reason python is not using homebrew's openssl algorithm
for finding certificates.

So I remain at a loss as to why Python standard library is not validating
certificates that are in the OpenSSL directory specified in the documents as
well as the Keychain (in both `.pem` and `.p12` formats, with "always trust"
for `Secure Sockets Layer (SSL)`).


Answer: This is a `dev_appserver` bug caused by a `httplib.HTTPSConnection` behavior
change (certificate check turned on by default) in some recent Python release
(I belive 2.7.9).

As the bug is in internal `dev_appserver` code (file
`google_appengine/google/appengine/api/urlfetch_stub.py` of the appengine SDK)
that is run independently of the tested application, there is no way to make a
fix that will survive a SDK update.

The only permanent workaround I can think of will be to enable
`validate_certificate` and add **CA** certificate to the
`urlfetch_cacerts.txt` file. As a temporary fix, you can patch
`urlfetch_stub.py` with workaround #3.

