title: How to maximise UDP packets-per-second with Go?
body: <p>I'm trying to write a simple relay server in Go for the purposes of sending game data between two clients. The clients send UDP registration messages consisting of the protocol id followed by their match id and their client id; These messages are used to set up a hashmap of addresses to address so that when a standard (game-data) message arrives we can map it to it's pre-registered recipient (without having to send that data in each message). (I'm using <a href="https://github.com/cornelk/hashmap" rel="nofollow noreferrer">github.com/cornelk/hashmap</a> if that makes any difference and the server is a highcpu-16 GCP Compute instance)</p>
  
  <p>This setup works fine with a low number of clients (~30 messages per second). However, as I ramp up the load testing the outgoing bandwidth levels off as incoming bandwidth continues to rise. I've done logging with <a href="http://man7.org/linux/man-pages/man8/vmstat.8.html" rel="nofollow noreferrer"><code>vmstat</code></a> and <a href="http://man7.org/linux/man-pages/man8/ifstat.8.html" rel="nofollow noreferrer"><code>ifstat</code></a> (as well as some <a href="https://github.com/nhorman/dropwatch" rel="nofollow noreferrer"><code>dropwatch</code></a> monitoring, which says lots of packets are being dropped at the software level). It looks to me like packets are being dropped because my go server isn't reading them quick enough.</p>
  
  <p>Initially, I used one goroutine per cpu core:</p>
  
  <pre><code>runtime.GOMAXPROCS(runtime.NumCPU())
  
  connection, err := net.ListenUDP("udp", &amp;addr)
  if err != nil {
      panic(err)
  }
  
  for i := 0; i &lt; runtime.NumCPU(); i++ {
      go listen(connection, c)
  }
  </code></pre>
  
  <p>I've also tried reusing the port and listening separately with each goroutine (using the <a href="https://github.com/libp2p/go-reuseport" rel="nofollow noreferrer">github.com/libp2p/go-reuseport</a> package).</p>
  
  <p>Finally, I tried setting up buffered channels for incoming and outgoing messages, in an effort to minimise the time that the listening goroutines aren't pulling messages.</p>
  
  <p>What am I missing in my attempt to handle 100K players at a time? It feels like no matter my approach I cannot surpass around 3000 players without losing a significant number of packets.</p>
  
  <p>My <code>listen</code> function has gone through many iterations in vain attempts to get the packet-per-second up but it's generally:</p>
  
  <pre><code>// Emits SIGABRT to the interrupts channel if an error occurs outside of individual message handling.
  func listen(connection *net.UDPConn, interrupts chan os.Signal/*, inbox chan IncomingMessage*/) {
      buffer := make([]byte, 1024)
      n, remoteAddr, err := 0, new(net.UDPAddr), error(nil)
      for err == nil {
          n, remoteAddr, err = connection.ReadFromUDP(buffer)
          if err != nil {
              continue
          }
  
          //log.Println("Received", n, "bytes", hex.EncodeToString(buffer[:n]))
          if n &lt; 2 {
              log.Println("Minimum packet length is 2 bytes. Received a packet of length", n, "bytes")
              continue
          }
  
          //inbox&lt;-IncomingMessage{
          //    sender: remoteAddr,
          //    data:   append([]byte(nil), buffer[:n]...),
          //}
  
          //go handlePacket(inbox, remoteAddr, append([]byte(nil), buffer[:n]...), n)
  
          protocolId := binary.LittleEndian.Uint16(buffer)
          if protocolId == registrationProtocolId {
              // Start a goroutine to handle the packet (copy the buffer minus the protocol id))
              //go handleRegistrationPacket(connection, remoteAddr, append([]byte(nil), buffer[:n]...))
              handleRegistrationPacket(outbox, remoteAddr, buffer[:n])
          } else if protocolId == matchProtocolId {
              //go handleStandardPacket(connection, remoteAddr.String(), append([]byte(nil), buffer[:n]...))
              handleStandardPacket(outbox, remoteAddr.String(), buffer[:n])
          } else {
              log.Println("Unrecognised protocol id: ", protocolId)
          }
      }
      log.Println("Listener failed:", err)
      interrupts&lt;-syscall.SIGABRT
  }
  </code></pre>
  
  <p>This graph shows 1600 concurrent games (3200 clients). Past a certain point, the outgoing KB/s stops climbing. The CPU isn't even breaking a sweat.
  <a href="https://i.stack.imgur.com/dFahA.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/dFahA.png" alt="Load-Test Graph showing Network bandwidth and cpu load"></a></p>