Multiple database connections class
Neat idea! Some suggestions:


__exit__ should try to commit or roll back based on whether there is an unhandled exception, should then close the cursor, and should finally unconditionally close the connection. This is one of the main reasons for using a context manager - it is expected to always clean up after itself, leaving the relevant parts of the system in the same state as before using it.
I would reuse the configuration section name as the keys in the dict. That way you don't need to maintain a mapping in your head or the code - what's in the configuration is what you get when you use the context manager.
Rather than opening all connections and then using only one of them, it should open the connection and the cursor with the name passed in. Otherwise you're wasting resources.