High power consumption of ATmega32u4 when connected to iOS device
The major difference between ATmega32u4 and ATmega328 is that the former has USB functionality built in, while the latter requires an external USB to UART chip. When such UART chips go through enumeration, they will tell the host that they only consume 100 mA or less, because they have no idea what else you have on the board. ATmega32u4 however can potentially consume up to 200 mA (sinking 100 mA through each of the IO banks). Maybe it's requesting that current from the host when you connect it, and your phone refuses to power up a device requesting more than 100 mA of current.

You can connect your controller to a computer and check USB properties to confirm this.