title: Fastest approach to using raw sockets in .NET
body: <p>I am trying to improve the performance of .NET 5 based network code to improve the throughput. I have been using the standard .NET Sockets implementation - but have hit something of a brick wall when it comes to throughput at around 200Mbps (1kb packets). I have tried both variants of async pattern (Begin... and SendToAsync...) and using a dedicated thread using synchronous operations - but can't improve performance any further. The overall CPU load is less than 50% on sender and receiver - and the machines are connected at 1Gbps.</p>
  <p>I have also tried using Pcap.Net and even a Windows Packet Filter driver (<a href="http://ntkernel.com" rel="nofollow noreferrer">http://ntkernel.com</a>) to plug into NDIS below the sockets layer.</p>
  <p>Reading around suggests that perhaps Registered I/O would be a good approach, but in my case I need to use SocketType.Raw (since I need to handle GRE, SCTP and various other protocols other than UDP and TCP). It appears that the Registered I/O API does not support raw (IP-level) sockets (see <a href="https://docs.microsoft.com/en-us/windows/win32/api/mswsock/nc-mswsock-lpfn_riocreaterequestqueue" rel="nofollow noreferrer">https://docs.microsoft.com/en-us/windows/win32/api/mswsock/nc-mswsock-lpfn_riocreaterequestqueue</a>)</p>
  <p>The profiler suggests that the time is spent within WSASendTo - and the .NET Socket class introduces little additional overhead. This suggests that even moving from C# to C++ using Winsock directly is unlikely to help much. I would rather steer clear of getting into Kernel development.</p>
  <p>Is there anything similar to Registered I/O which would work with raw sockets? It seems strange that Registered I/O should support UDP but not IP directly. Any other ideas how I can resolve this bottleneck? Are there any inefficiencies inherent in Winsock relating to raw sockets (as opposed to UDP and TCP)?</p>
  <p>As requested - added a simple example to send 1Kb dummy GRE packets as fast as possible. This results in about 270Mbps throughput and about 30% CPU evenly across all cores. Performance is broadly the same using localhost - which I think rules-out the network card as being the bottleneck.</p>
  <pre><code>using System;
  using System.Collections.Concurrent;
  using System.Net;
  using System.Net.Sockets;
  
  namespace RawSender
  {
      class Program
      {
          private static readonly ConcurrentBag&lt;SocketAsyncEventArgs&gt; ArgsPool = new ConcurrentBag&lt;SocketAsyncEventArgs&gt;();
          static void Main(string[] args)
          {
              var Destination = IPAddress.Parse(args[0]);
  
              // Create pool of SocketAsyncEventArgs for the async operations
  
              for (int Index=0;Index&lt;10_000;Index++)
              {
                  var Args = new SocketAsyncEventArgs();
                  Args.SetBuffer(new byte[1000]);
                  Args.Completed += OnComplete;
                  Args.RemoteEndPoint = new IPEndPoint(Destination, 0);
                  ArgsPool.Add(Args);
              }
  
              // Send 1 million dummy 1kb packets using IP protocol 47 (GRE)
              
              var Socket = new Socket(AddressFamily.InterNetwork, SocketType.Raw, (ProtocolType)47);
              for(int Scan=0;Scan&lt;1_000_000;Scan++)
              {
                  if (!ArgsPool.TryTake(out SocketAsyncEventArgs Args)) throw new Exception(&quot;Args pool empty&quot;);
                  if (!Socket.SendToAsync(Args))
                  {
                      OnComplete(null, Args);
                  }
              }
          }
  
          private static void OnComplete(object sender, SocketAsyncEventArgs e)
          {
              ArgsPool.Add(e);
          }
      }
  }
  </code></pre>