title: What can be the reason for CPU load to NOT scale linearly with the number of traffic processing workers?
body: <p>We are writing a Front End that is supposed to process large volume of traffic (in our case it is Diameter traffic, but that may be irrelevant to the question). As client connects, the server socket gets assigned to one of the Worker processes that perform all the actual traffic processing. In other words, Worker does all the work, and more Workers should be added when more clients get connected.</p>
  
  <p>One would expect the CPU load per message to be the same for different number of Workers, because Workers are totally independent, and serve different sets of client connections. Yet our tests show that it takes more CPU time per message, as the number of Workers grow. </p>
  
  <p>To be more precise, the CPU load depends on the TPS (Transactions or Request-Responses per second) as follows.</p>
  
  <p>For 1 Worker:
  60K TPS - 16%, 65K TPS - 17%... i.e. ~0.26% CPU per KTPS</p>
  
  <p>For 2 Workers: 
  80K TPS - 29%, 85K TPS - 30%... i.e. ~0.35% CPU per KTPS</p>
  
  <p>For 4 Workers:
  85K TPS - 33%, 90K TPS - 37%... i.e. ~0.41% CPU per KTPS</p>
  
  <p>What is the explanation for this? Workers are independent processes and there is no inter-process communication between them. Also each Worker is single-threaded.</p>
  
  <p>The programming language is C++
  This effect is observed on any hardware, which is close to this one: 2 Intel Xeon CPU, 4-6 cores, 2-3 GHz
  OS: RedHat Linux (RHEL) 5.8, 6.4
  CPU load measurements are done using mpstat and top.</p>