MIDI output circuit behaving incorrectly when not connecting with USB
If you are connecting multiple devices then they have to share a common ground.  

I only know about British mains sockets, they all have an earth pin connecting to a common ground, so usually you are OK.

Plug in transformers 'wall warts' sometimes don't connect to ground, the give away is the plastic pin.  In this case you need to provides a separate earth and connect it to the rest of the equipment.

PCs normally have a good earth and this is passed by the USB to connected devices.  So "the Arduino always works when it is connected to the PC" is usually a good indicator of an earthing issue.

I think your keyboard uses a wall wart and therefore when you remove the USB they both float.  Touching the USB lead to the Arduinos USB socket grounds it all again.

So it looks like you diagnosed the fault correctly :)