How to protect a device which has different levels of access to its storage? There is a device which has three software running on it, the main purpose is the control of some machinery. 


a GUI front-end, running on a single-board computer with easily accessible storage (hard disk or SD card, etc.)
a back-end, running on the same SBC as the above.
a microcontroller which drives the hardware, and communicates solely with the back-end. The messages between #2 and #3 are short, in the range of 10 to at most 50 bytes.


Income is solely made by manufacturing and selling the device itself. However, the majority of the development costs are made up by developing the software #1 and #2.

The threat model is someone making a physical clone of the device, implementing their own version of software #3, and stealing software #1 and #2. For our threat model, for the attacker to read the memory (either volatile or non-volatile) of the microcontroller in #3 is to be considered impossible. Wiretapping the communication between #2 and #3 is to be considered easy.

This looks like a standard case of software protection dongle, with its obvious drawbacks: if it's only "check for the presence of the dongle", even if it's based on sending randomly generated code encrypted by the public key on one end, and decrypting it with the private key on the other end, it's easy to defeat with a single change in a single jump instruction to bypass the "if". Anything more would require very complicated solutions, like executing some code meant to be executed in #1 or #2 in the microcontroller instead.

Instead, I'm considering the following:


The microcontroller, obviously, will hold the private key of an asymmetric encryption.
The commands sent from #2 to #3 will be encrypted by the public key.
The replies (mainly status messages about the states of the controlled machinery) sent from #3 to #2 will be encrypted with the private key.


Does this scheme has critical drawbacks? Are there better alternatives?

I can see the following problems:


It doesn't secure #1 at all. (Maybe it could be done by routing it through #2 but it would make it needlessly complicated)
There are not too many types of messages and the messages are relatively short. Especially replies could be attacked, as the state of the machinery is visible, so by wiretapping one can see what kind of message gets generated in which state of the machinery. Yes, the messages could be padded, but not by orders of magnitude (it would reduce bandwidth), and adding random junk seems to be like a form of security through obscurity.
This question may be more suitable for other forums (https://security.stackexchange.com/ ?) This is not about cryptography.

I don't think cryptography will help you to secure the devices to protect your IP to the extend you'd like. Once you hand over the hardware to the 3rd party, there isn't much you can do. 

Please do not count this as an answer, as it won't help you to achieve your goals. This "answer" should help you to clarify some options 


  stealing software #1 and #2...
  change in a single jump instruction to bypass the "if"


So you are already assuming the SBC running the frontend could be fully read and the software modified (this is often so). If you give the adversary this power, you cannot do much about it. In that case "obfuscation" is the best what I can see (any better idea?). 


  or the attacker to read the memory (either volatile or non-volatile) of the microcontroller in #3 is to be considered impossible


That's a very strong assumption.. though you've linked a "secure microprocessor", maybe you've already have something like that in mind  (smart card with crypto features? hms?)


  The commands sent from #2 to #3 will be encrypted by the public key.
  The replies (mainly status messages about the states of the controlled  machinery) sent from #3 to #2 will be encrypted with the private key.


Encryption is done with public keys, you could decrypt with a private key. So the microcontroller and the frontend both need each its own keypairs (private and public). PKI encryption will blow the message a lot, it is very performance expensive.

However - when you already came to an assumption that the adversary could read everything off the frontend (including the private key), the adversary could inpersonate the frontend and read/decrypt the messages anyway (accessing the private key of the frontend).