How to protect a device which has different levels of access to its storage?
This question may be more suitable for other forums (https://security.stackexchange.com/ ?) This is not about cryptography.

I don't think cryptography will help you to secure the devices to protect your IP to the extend you'd like. Once you hand over the hardware to the 3rd party, there isn't much you can do. 

Please do not count this as an answer, as it won't help you to achieve your goals. This "answer" should help you to clarify some options 


  stealing software #1 and #2...
  change in a single jump instruction to bypass the "if"


So you are already assuming the SBC running the frontend could be fully read and the software modified (this is often so). If you give the adversary this power, you cannot do much about it. In that case "obfuscation" is the best what I can see (any better idea?). 


  or the attacker to read the memory (either volatile or non-volatile) of the microcontroller in #3 is to be considered impossible


That's a very strong assumption.. though you've linked a "secure microprocessor", maybe you've already have something like that in mind  (smart card with crypto features? hms?)


  The commands sent from #2 to #3 will be encrypted by the public key.
  The replies (mainly status messages about the states of the controlled  machinery) sent from #3 to #2 will be encrypted with the private key.


Encryption is done with public keys, you could decrypt with a private key. So the microcontroller and the frontend both need each its own keypairs (private and public). PKI encryption will blow the message a lot, it is very performance expensive.

However - when you already came to an assumption that the adversary could read everything off the frontend (including the private key), the adversary could inpersonate the frontend and read/decrypt the messages anyway (accessing the private key of the frontend).