Lightweight point to point communication protocols
Why do you not intend to use an existing protocol?

AFAICT, there are a bunch of protocols already in use, developed over many years, that have solved many of the problems, based on research and experience. So I'd use something that works and has been debugged unless you are doing research. If it is research, you need to describe the complete use-case, and not just some specific features of the use-case.

Old protocols may survive simply because the technology in satellites not only has to work, but also needs to be tested, characterised and 'certified' to work in space. That is an expensive, specialised and slow process. So saving cost and time by using something that is already known to work reliably makes even more sense than the usual Earth-bound engineering projects.

Is this for an commercial, research, or amateur project? 

I know the amateurs have built the equivalent of a global network of ground stations, which hand-off to the next station as the satellite orbits (a bit like the ground-station network NASA or other space agencies have, but using equipment made available for parts of a day [the owners of the equipment use it too, themselves, for their own purposes] by enthusiastic amateurs). Are you trying to solve that problem? 

In which case it might not be a single point-to-point communication. It may be multiple Earth-based stations each trying to talk to the same satellite. Or it may be multiple Earth-based stations trying to talk to multiple satellites.

If it is an amateur project, contact groups directly. The UK Amateur Satellite group is friendly, and members collaborated with colleagues in AMSAT-NL, in the creation of the software and network for the amateur-radio-based international network of ground-stations. They also built the low-cost radio hardware used to track their FUNCube CubeSat.

One of the earliest packet wireless networks ALOHnet went on to influence Ethernet and Inmarsat. So, for example, some of the successful ideas may be retained from that work. As long as it works, has no significant disadvantages, has been proven to work in space, and has hardware available, which is tested and space 'certified', what motivation is there to change?

Edit:  

There are some papers on high-latency, and even a stackoverflow question Networking with extremely high latency which might give some inspiration.

There are old protocols with debugged implementations. These have the benefit that they are so old the machines they ran on had few resources, and so may be well suited to your application. 

For example, the Kermit Protocol. That wikipedia article says "Kermit software has been used for tasks ranging from simple student assignments to solving compatibility problems aboard the International Space Station." So it has some useful pedigree :-)

It supports a 'sliding window protocol' which supports selective retransmission. That should help recover from errors and do re-assembly of damaged streams of data. 

It is Open Source, and had C implementations. So you might be able to get the source and port it. Kermit ran on machines with less than 64k address space. So it might fit onto your hardware.

Even if Kermit is not a good fit, I would recommend looking at old (circa 1980's) protocols and implementations because the machine constraints may be comparable to the constraints of your in-flight hardware; even if your hardware is less constrained than the 1980's, your power budget might be helped by a low-resource protocol implementation.