title: High-performance TCP Socket programming in .NET C#
body: <p>I know this topic is already asked sometimes, and I have read almost all threads and comments, but I'm still not finding the answer to my problem.</p>
  <p>I'm working on a high-performance network library that must have TCP server and client, has to be able to accept even 30000+ connections, and the throughput has to be as high as possible.</p>
  <p><strong>I know very well I have to use <code>async</code> methods</strong>, and I have already implemented all kinds of solutions that I have found and tested them.</p>
  <p>In my benchmarking, only the minimal code was used to avoid any overhead in the scope, I have used profiling to minimize the CPU load, <strong>there is no more room for simple optimization</strong>, on the receiving socket the buffer data was always read, counted and discarded to avoid socket buffer fill completely.</p>
  <p>The case is very simple, one TCP Socket listens on localhost, another TCP Socket connects to the listening socket <em>(from the same program, on the same machine oc.)</em>, then one infinite loop starts to send <strong>256kB</strong> sized packets with the client socket to the server socket.</p>
  <p>A timer with 1000ms interval prints a byte counter from both sockets to the console to make the bandwidth visible then resets them for the next measurement.</p>
  <p>I've realized the sweet-spot <strong>for packet size is 256kB</strong> and the <strong>socket's buffer size is 64kB</strong> to have the maximum throughput.</p>
  <p>With the <strong><code>async/await</code></strong> type methods I could reach</p>
  <pre><code>~370MB/s (~3.2gbps) on Windows, ~680MB/s (~5.8gbps) on Linux with mono
  </code></pre>
  <p>With the <strong><code>BeginReceive/EndReceive/BeginSend/EndSend</code></strong> type methods I could reach</p>
  <pre><code>~580MB/s (~5.0gbps) on Windows, ~9GB/s (~77.3gbps) on Linux with mono
  </code></pre>
  <p>With the <strong><code>SocketAsyncEventArgs/ReceiveAsync/SendAsync</code></strong> type methods I could reach</p>
  <pre><code>~1.4GB/s (~12gbps) on Windows, ~1.1GB/s (~9.4gbps) on Linux with mono
  </code></pre>
  <p>Problems are the following:</p>
  <ol>
  <li><code>async/await</code> methods were the <strong>slowest</strong>, so I will not work with them</li>
  <li><code>BeginReceive/EndReceive</code> methods started new async thread together with the <code>BeginAccept/EndAccept</code> methods, under Linux/mono <strong>every new instance of the socket was extremely slow</strong> (when there was no more thread in the <code>ThreadPool</code> mono started up new threads, but to create <strong>25 instance</strong> of connections did take about <strong>5 mins</strong>, creating <strong>50 connections was impossible</strong> (program just stopped doing anything after ~30 connections).</li>
  <li>Changing the <code>ThreadPool</code> size did not help at all, and I would not change it (it was just a debug move)</li>
  <li>The best solution so far is <code>SocketAsyncEventArgs</code>, and that makes the highest throughput on Windows, but in Linux/mono it is slower than the Windows, and it was the opposite before.</li>
  </ol>
  <p>I've benchmarked both my Windows and Linux machine with <a href="https://iperf.fr/" rel="noreferrer">iperf</a>,</p>
  <pre><code>Windows machine produced ~1GB/s (~8.58gbps), Linux machine produced ~8.5GB/s (~73.0gbps)
  </code></pre>
  <p>The weird thing is <strong><code>iperf</code></strong> could make a weaker result than my application, but on Linux, it is much higher.</p>
  <p>First of all, I would like to know if the results are normal, or can I get better results with a different solution?</p>
  <p>If I decide to use the <code>BeginReceive/EndReceive</code> methods (they produced relatively the highest result on Linux/mono) then how can I fix the threading problem, to make the connection instance creating fast, and eliminate the stalled state after creating multiple instances?</p>
  <p>I continue making further benchmarks and will share the results if there is any new.</p>
  <p>================================= UPDATE ==================================</p>
  <p>I promised code snippets, but after many hours of experimenting the overall code is kind of a mess, so I would just share my experience in case it can help someone.</p>
  <p>I had to realize <strong>under Window 7 the loopback device is slow</strong>, could not get higher result than 1GB/s with <a href="https://iperf.fr/" rel="noreferrer">iperf</a> or <a href="https://gallery.technet.microsoft.com/NTttcp-Version-528-Now-f8b12769" rel="noreferrer">NTttcp</a>, <strong>only Windows 8 and newer versions have fast loopback</strong>, so I don't care anymore about Windows results until I can test on newer version. <a href="https://msdn.microsoft.com/en-us/library/windows/desktop/jj841212(v=vs.85).aspx" rel="noreferrer">SIO_LOOPBACK_FAST_PATH</a> should be enabled via <a href="https://docs.microsoft.com/en-us/dotnet/api/system.net.sockets.socket.iocontrol?redirectedfrom=MSDN&amp;view=netframework-4.7.2#System_Net_Sockets_Socket_IOControl_System_Net_Sockets_IOControlCode_System_Byte___System_Byte___" rel="noreferrer">Socket.IOControl</a>, but it throws exception on Windows 7.</p>
  <p><strong>It turned out the most powerful solution is the Completion event based <a href="https://code.msdn.microsoft.com/windowsdesktop/High-Performance-NET-69c2df2f" rel="noreferrer">SocketAsyncEventArgs</a> implementation both on Windows and Linux/Mono.</strong> Creating a few thousand instances of the clients never messed up the ThreadPool, the program did not stop suddenly as I mentioned above. This implementation is very nice to the threading.</p>
  <p>Creating 10 connections to the listening socket and feeding data from 10 separate thread from the <code>ThreadPool</code> with the clients together could produce <code>~2GB/s</code> data traffic on Windows, and <code>~6GB/s</code> on Linux/Mono.</p>
  <p>Increasing the client connection count did not improve the overall throughput, but the total traffic became distributed among the connections, this might be because the CPU load was 100% on all cores/threads even with 5, 10 or 200 clients.</p>
  <p>I think overall performance is not bad, 100 clients could produce around <code>~500mbit/s</code> traffic each. <em>(Of course this is measured in local connections, real life scenario on network would be different.)</em></p>
  <p>The only observation I would share: experimenting with both the Socket in/out buffer sizes and with the program read/write buffer sizes/loop cycles highly affected the performance and very differently on Windows and on Linux/Mono.</p>
  <p><strong>On Windows</strong> the best performance has been reached with <code>128kB socket-receive</code>, <code>32kB socket-send</code>, <code>16kB program-read</code> and <code>64kB program-write</code> buffers.</p>
  <p><strong>On Linux</strong> the previous settings produced very weak performance, but 512kB <code>socket-receive and -send</code> both, <code>256kB program-read</code> and <code>128kB program-write</code> buffer sizes worked the best.</p>
  <p>Now my only problem is if I try create 10000 connecting sockets, after around 7005 it just stops creating the instances, does not throw any exceptions, and the program is running as there was no any problem, but I don't know how can it quit from a specific <code>for</code> loop without <code>break</code>, but it does.</p>
  <p>Any help would be appreciated regarding anything I was talking about!</p>