Securing communication between limited embedded systems I am currently trying to harden communication over a limited physical communication channel on an embedded system.
The Scenario
I think the system is rather typical for small embedded applications:

The embedded systems involved are small microcontrollers without any OS, which
can be assumed to be in a secure physical environment. The connection
channel is physically accessible to aggressors, they can interfere
with messages as desired.

Different communication channels are possible: e.g. RS485 or RS232.
There is a crude protocol which handles stuff like addressing and crude
integrity (e.g. CRC16 for bitflips) on which we can build. Think of
it as being a slow TCP.

Each communication partner knows a root certificate which can
act as a CA and has an own unique certificate(+key) signed by it. Certificates are self signed, elliptic curve prime256v1

The mbedTLS library is available for these systems (with DH, AES, ..).

Most of the time there is no internet available.

The systems can keep time and can create sufficient random numbers


I want to reach privacy, authenticity and integrity.
How it might be solved
A rule in cryptographic protocols is to never implement them on your own.
Unfortunately I did not find anything that matches this use case.
I also browsed through existing questions, but nothing provided the full solution i was searching for.
Finally in this answer implementing an own protocol is given as the last option.
This is why I came up with the solution described below - but I am not entirely sure about it.
I oriented on this answer on how to authenticate a Diffie-Hellmann Key Exchange, this answer on how to check integrity and this thread giving me some ideas on how to implement such a protocol.
Protocol to secure communication
First thing to do is to randomly generate a symmetric key K.
First thing sent is a handshake message.
It has the following fields:




Field
Description




DH
The Diffie-Hellmann key exchange msg gK


Cert
The certificate(-chain) of the sending device


MsgIdNonce
A random 64-bit value


Signature
A signature of the complete message using the private key of Cert




Both devices send (and receive) such a message.
When a message is received the following is done:

Check if cert is signed by the expected CA (has to be in chain)
Check if msg is properly signed with cert and its key

When these checks succeed the sent and received DH gK messages are used to calculate a symmetric key.
From this point on, applications can send messages encrypted with AES-CBC. As messages could be lost relatively often, IV is always in the message.
This is the message format:




Field
Description
Encrypted




IV
Initialy Random IV for AES
No


MsgIdNonce
Handshake/Last MsgIdNonce + 1 or + 2
Yes


Application Data
The data/commands to protect
Yes


CMAC truncated to 64 Bit
MAC for checking integrity
Yes




For a message to be accepted it must

Have the expected MsgIdNonce (1 or 2 above last received one) - preventing replay attacks, allowing to lose 1 message
Have a valid CMAC - preventing attacks on integrity
Be received max. T seconds after the last message - preventing delay attacks

If anything is not as expected, Handshake is sent again, then message is sent again with newly exchanged parameters.
Questions
Am I on the right path or should I try something completely different?
Does this provide my security goals or do I miss any obvious vulnerabilities here?
Can this work?
I hope you can help me and other embedded engineers facing similar problems!
A rule in cryptographic protocols is to never implement them on your own. Unfortunately I did not find anything that matches this use case.

Have you considered DTLS?  That tries to address the same problem (except for time stamps; that could be added by inserting the time stamp in the datagram), and already has things thought through.  Another possibility would be IPsec; however DTLS looks closer to addressing the problem you have (unless the data traffic you're protecting is IP traffic).
That said, here are some thoughts on the protocol you outlined:

What happens if two sequential messages are dropped (e.g. received incorrectly)?  Will the two sides realize what happened, or will they proceed to drop all subsequent messages (because of the 'nonce has to increment no more than 2' rule)?

Presuming they attempt a rekey, how is this coordinated?  If one side thinks they haven't rekeyed yet, and the other side thinks they are, what will happen to messages encrypted with the old key?  We find it generally useful to include a 'which key this is encrypted under' tag with the ciphertext (in DTLS, this is the 'epoch').

Can your communication channel reorder messages?  I suspect it can't in your case, but if it can, you should think about how that should be handled.

What happens if someone injects a previous (valid) handshake message?  How will that confuse things?

In the data encryption path, you use CBC encryption and CMAC integrity.  While this is a workable solution, this is prone to various padding attacks if they aren't integrated correctly.  Current fashion is to use an AEAD mode (such as GCM or Chacha20/Poly1305) which does both.

Is the encrypted traffic unidirectional or bidirectional?  If traffic goes both ways, do you use the same set of keys to protect both sets of traffic?

&quot;If anything is not as expected, Handshake is sent again, then message is sent again with newly exchanged parameters.&quot; - how does that work?  If the receiver decided it didn't like the message, how does the sender know to resend it?