title: UDP server app on Linux can't receive packets from more than 150 clients
body: <p>I've written a UDP C++ server app on Linux and am now load testing it to see how many clients it can handle.  I find that it peaks at about 150 simultaneous clients sending packets at a rate of 2-4 per second.</p>
  
  <p>Clients added after that will cause some other clients' packets to be dropped.</p>
  
  <p>The server itself is not stressed, using less than 10% of the CPU and memory.  The network is not stressed at all either, at about 15K bytes/second.  The packets are arriving at the server (which uses one UDP socket for both read and write) at about 200 packets/second.  The server threads themselves still sleep for short periods at this load level.</p>
  
  <p><strong>Question</strong>:<br>
  Any ideas on what is the bottleneck here?  CPU, network and server code itself all seem to be unstressed.  Will the OS be unable to handle this # of UDP packets?</p>
  
  <p>The hardware is very low power - a single core Pentium equivalent at 1.5 MHz. The NIC is 100M bits/second.  I'm running Ubuntu 11.1.</p>
  
  <p>This article might be related: <a href="https://stackoverflow.com/questions/10544296/upper-limit-to-udp-performance-on-windows-server-2008">Upper limit to UDP performance on windows server 2008</a></p>
  
  <p><em>Update</em>:
  The server sets up a UDP socket, then creates 3 threads and 2 queues.  The first thread blocks on socket read and looks like:</p>
  
  <pre><code>while (1)
  {
      recvfrom(this-&gt;socket, readBuf, BUFSIZE, 0, (sockaddr *)&amp;address, &amp;addressLen);
      pushBack(this-&gt;inputQueue, message);
  }
  </code></pre>
  
  <p>The second thread sleeps on the inputQueue.  It wakes up when a condition is signaled and processes messages.  It sends processed messages to an outputQueue:</p>
  
  <pre><code>while (1)
  {
      sleepOnQ(this-&gt;inputQueue);
      popFront(this-&gt;inputQueue);
      processMessage();
      pushBack(this-&gt;outputQueue, message);
  }
  </code></pre>
  
  <p>The 3rd thread sleeps on the outputQueue and sends messages out the UDP socket to the destination.  Note, it's the same socket that's used for reading.</p>
  
  <pre><code>while (1)
  {
      sleepOnQ(this-&gt;outputQueue);
      popFront(this-&gt;outputQueue);
      processMessage();
      sendto(this-&gt;socket, message, ... );
  }
  </code></pre>
  
  <p>The amount of processing per client and message is small.  As I mentioned when the server is handling 200 messages/second, it's using about 10% of a really wimpy CPU.</p>
  
  <p>Here are some of the kernel parameters on the system:</p>
  
  <pre><code>net.core.wmem_max = 114688
  net.core.rmem_max = 114688
  net.core.wmem_default = 114688
  net.core.rmem_default = 114688
  </code></pre>
  
  <p><strong>More info on data synchronization</strong></p>
  
  <p>The answers so far have made me think two things are going on:</p>
  
  <ol>
  <li>the OS read buffer is filling up.  But with low CPU this shouldn't happen</li>
  <li>But #1 can happen if the threads are waiting for other events and because of that don't read the socket quickly enough.</li>
  </ol>
  
  <p>Logging could be an issue, I'll try turning it off and report results.  However, probably more important is the contention for the queues between threads.  Since CPU is low, maybe the threads spend a lot of time waiting for access to the queues.</p>
  
  <p>In the first iteration of this server I tried to be tricky about locking data.  The server was very fast but crashed when it got to 800 packets/second.  The current version locks the entire queue.  Maybe I need a better way to synchronize the threads.</p>
  
  <p><strong>Question answered</strong></p>
  
  <p>The information I got here was very helpful.  The problem was a bonehead error with the test client, but doing the investigation helped me eliminate the cause suggested here.  </p>
  
  <p>FYI here are my results.  Once I fixed the problem with the client the server accepted about 800 packets/second with 70% CPU utilization.  I had increased the OS read/write buffer to 12MB from 128K.  I didn't test whether read buffer was filled.  I doubt the OS read buffer was an issue because at top speed the server read thread was still blocking on read for short periods every 10th or 20th read.</p>
  
  <p>800 packets/second is still too slow, so I removed logging from the server.  This made a huge difference.  Server was able to receive 2900 messages/second from 1400+ clients at 70% cpu util.</p>
  
  <p>I also did some testing with whether the read thread was waiting for the lock.  Even at top speed I found it never had to wait more than 1 ms, so it wasn't a factor at 2900 messages/second.  Perhaps it will be on a faster CPU.</p>
  
  <p>At this point the server is CPU bound and to find the next bottleneck I'll need to get on a more powerful CPU.
  Thanks for your help!</p>