Serial communication best practices
I would suggest a few principles which, while not required, may save you some grief:


Define your protocol in such a way that any time some particular byte sequence is seen, it will--with no exceptions--mark the beginning of a packet.  In all your receiving code, watch for that sequence regardless of anything else that might be going on.
Having a byte sequence which always indicates "start of packet" will generally make it necessary to "escape" data which might naturally contain that byte sequence.  It's not necessary to do anything absurdly complicated in designing one's escape-sequence logic, but it may be desirable to put a cap on worst-case performance.  For example, if one used the character sequence ~S (7E 53) to indicate start of packet, and replaced any tildes that occurred within the data stream with ~T (7E 54), that would work, but data streams that contained lots of tildes could almost double in length.  One could cap the worst-case behavior at 50% growth by replacing all occurrences of two consecutive tildes in the data stream with ~D.  Note that I would recommend against what I consider the all-to-common practice of doubling the escape character to indicate itself.  If ~S indicated start of packet and ~~ was a literal tilde, code which received the sequence ~~S wouldn't be able to tell whether it was an escaped tilde followed by an S, or was a tilde from a truncated packet followed by a start-of-packet indicator.
Only one side of the communication should care about timing, and each message from that side should yield a response by the other.  If the master asks the slave for information and the slave sends it, the slave shouldn't worry about whether the master gets it; if the master doesn't get the information again, it should re-request it.  If the master doesn't re-request the information, the slave should figure it doesn't need to re-send it.
If possible, the protocol should be designed so that sending a command multiple times will have the same effect as sending it once.  This may require adding sequence numbers to commands, or splitting commands into multiple parts.
Be aware of the medium over which data will be sent.  Some communications media (including some radio links) can pretty well guarantee that if something is sent and not received within a short time, it will never be received.  Other media (e.g. TCP sockets, or some other radio links) offer no such promise.  If a TCP connection remains open, all data will be received in the order sent, though perhaps with some delay; some "meshed" radio links may try over a period of seconds to send messages, and messages which are delivered via different routes may arrive out of sequence.
I almost always include an "echo" command in the protocols I design.  Not only is such a thing simple, and not only can it be a useful diagnostic tool for identifying whether "anything" is working, but the ability to send an echo command with a random payload and listen for the response is very helpful with things like TCP connections which can have significant delays (if the master sends a random payload and gets that same payload back, it can be sure that as long as it receives a response to each command, every response will be a reply to the command that immediately preceded it).  Absent such synchronization, one may end up in a situation where after sending each command, one mistakes misinterprets the response from the previous command as being a response to the latest one.