Authenticated Encryption of UDP packets I am working on a UDP based file transfer protocol. Congestion Control, Resending of packages and other UDP challenges are dealt with - speedtests shows that the protocol is up to 10 times faster than e.g. HTTP for transferring large files when the latency/rtt is > 100 ms (e.g. between Europe and the US).

The protocol will be used in a server/client architecture - where a server will send/receive data to/from many clients. 

Servers are running .NET - and clients will be written in .NET, Java (Android), and Swift (OS X, iOS). 

And - it's now time to make the UDP protocol secure.

Some requirements:


Must be secure and fast :)
Must be supported on .NET, OS X and Java
Must support some kind of SessionID - so that the server can separate different clients from another and use the appropriate key for each client.


Reading this post I agree with the approved answer by Otus:


Don't roll your own protocol, but use (D)TLS.
If you must roll your own protocol, use an authenticated encryption mode.
If you can't, at least use standard encryption and MAC.


Here are my thoughts regarding these 3 alternatives:

DTLS
The reason I am a bit sceptical towards DTLS is that I am unable to find any good/trusted implementations for it in C#. BouncyCastle is a museum of old crypto and bad ports according to  Inferno. Eldos has another implementation, but I am unsure of the quality + it's a bit pricey.

Authenticated Encryption Mode
AES-GCM is a good AEAD candidate, and also fast. Inferno has a .NET implementation and there seems to be good candidates for Java and OS X /iOS we well (OpenSSL). However - I'm a bit sceptical to getting all these libraries to work together correctly - just a gut feeling. SessionID could be included as AdditionalData (AEAD).

Standard Encryption and MAC - here's what I'm currently thinking:


Get TLS1.2 Cert Client retrieves the TLS1.2 certificate of the server through a .NET HTTPS call. The client has also authenticated the server by making this call.
Handshake. Client generates a KEY - and encrypts this with the public key of the server. The client sends a "handshake" UDP package to the server containing:


SessionID - A Guid generated by the client. Used to identify client for all subsequent communication. (I do not wish to use a combination of IP/Port to identify client - due to the parking-lot problem)
A SHA-256 Hash of (1) + (3) + KEY (often referred to as MAC)
The Generated KEY - AES-256 encrypted with the public key of the server.

The server and client now has a shared private KEY that they both use to synchronously encrypt/decrypt messages that they send/receive - very similar to the way SSL/TLS works I guess. (Doing async encryption/decryption with the private/public key of the server will be extremely slow) 
Each subsequent UDP package sent by the Server will contain:


SessionID - GUID. Plain text. Server uses this to lookup key on its side.
A SHA-256 Hash of (1) + (3) + KEY. This way - the plaintext SessionID is authenticated.
Payload encrypted with AES-256



The advantages - as I see them - are that I am using standardized (and relatively simple) building blocks that are well-proven and easy to implement/test/debug on all platforms. I also get an authenticated SessionID into the mix.

What are your thoughts? Is 3. a feasible option - or should I abandon this approach?
You can use DTLS here.  C# can call any C API via P/Invoke.  OpenSSL, BoringSSL, Network Security Services (NSS), and mbedtls all provide C interfaces to DTLS.

Unless there are existing bindings, you will need to implement at least some of the P/Invokes yourself, but this is much easier than writing your own cryptographic protocol.

It would be very nice if you could open-source these bindings, for the benefit of those who follow in your footsteps.

Also, note that AES-GCM is, in the absence of hardware support, very tricky to implement in a way that is immune to cache-timing side-channel attacks &mdash; so tricky, in fact, that many production implementations are not hardened against such attacks.  Implementations that are immune to cache-timing attacks deliver suboptimal performance.  ChaCha20-Poly1305 is an efficient, secure, and portable alternative.  It is several times faster than pure-software AES-GCM while being immune to many side-channel attacks.  Since you mention mobile devices, you probably will not have hardware support for AES, so ChaCha20 is both faster and easier to implement securely.