Where to store sensitive keys in a device? I am working on some security features that should be executed by the firmware of a consumer device whose communication to a server must be secure.

Any exchange of data is based on symmetric cryptography, with keys created by "Elliptic Curve Diffie Hellman Key Agreement".

For reasons of cost, a dedicated hardware security module (HSM) was omitted. Instead, all cryptographic algorithms are executed within the board firmware, additionally supported by powerful hardware accelerators directly on the MCU to handle AES and ECC. 


All RAM and used flash memory is internal to the MCU.
JTAG interface is disabled. 
No external memory is used. 
Beside of executing directly within the firmware, all cryptographic algorithms are state of the art.


The open question is, where to store sensitive keys: 

Is it acceptable from a security point of view, to keep private and symmetric keys in the MCU's flash memory, since there is no way to penetrate from outside, except perhaps the monitoring of power consumption as a side channel attack? To be more secure, all these keys are additionally encrypted by a unique device key, to be created upon factory startup. However, I feel, because this device key is also kept solely within internal flash that this gives only very little additional security.

What considerations should be considered to verify whether such an approach will fulfill standard security requirements or not? What would be the right way without using a dedicated HSM?
If it can be stored, it can be recovered. 

Consider the unsettling fact that even if you were able to select a hard-to-discover location, since you are using a consumer device, at any point in the supply chain (HDL design, whoever makes your microcontrollers, etc) someone could have compromised the device long before you get to designing the firmware. You are wise to observe the likely-negligible extra security obtained.

As others have mentioned, reactivating JTAG or UART ports is not that big of a deal, and I can just crack open your device and dump out the flash memory anyway. Understanding firmware is easier than ever. You should assume your adversary has access to IDA Pro, understands firmware emulation, and is willing to drop some cash on an EEPROM or flash reader. (This is what I do, anyway.) Being clever may slow an adversary down but it can provide no guarantees.

Even HSMs are not necessarily secure. It's turtles all the way down.