title: Improving Performance of Shared Memory - Isolating Linux Core
body: <p>I'm trying to optimize the performance of reading and writing a <code>double</code> to shared memory. I have one program writing to shared memory, and another reading from it. </p>
  
  <p>I've used <a href="https://stackoverflow.com/questions/13583146/whole-one-core-dedicated-to-single-process">this post</a> to help isolate CPUs for these two programs to run on, with the following line in my <code>etc/default/grub</code> file:</p>
  
  <pre><code>GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_idle.max_cstate=1 isolcpus=6,7"
  </code></pre>
  
  <p>I am using <code>taskset -c 6 writer</code> and <code>taskset -c 7 reader</code> to set these programs to run on these cpus.</p>
  
  <p>Using <a href="http://man7.org/linux/man-pages/man2/sched_setscheduler.2.html" rel="nofollow noreferrer">this man page on sched_setscheduler</a>, I have set up both programs to have the highest scheduling priority using the following code:</p>
  
  <pre><code>struct sched_param param;   
  param.sched_priority = sched_get_priority_max(SCHED_FIFO);
  
  if(sched_setscheduler(0, SCHED_FIFO, &amp;param) == -1) 
  {
       perror("sched_setscheduler failed");
       exit(-1);
  }
  </code></pre>
  
  <p>I have defined a struct to be used in shared memory that contains the required synchronization tools, as well as a timespec struct and a double to pass between the two programs, as follows:</p>
  
  <pre><code>typedef struct
  {
      // Synchronization objects
      pthread_mutex_t ipc_mutex;
      sem_t ipc_sem;
      // Shared data
      double value;
      volatile int read_cond;
      volatile int end_cond;
      double start_time;
      struct timespec ts;
  } shared_data_t;
  </code></pre>
  
  <p><strong>Shared Memory Initialization:</strong></p>
  
  <p><strong>Writer:</strong></p>
  
  <pre><code>// ftok to generate unique key 
  key_t key = ftok("shmfile",65); 
  
  // shmget returns an identifier in shmid 
  int shmid = shmget(key,1024,0666|IPC_CREAT); 
  ftruncate(shmid, sizeof(shared_data_t));
  
  // shmat to attach to shared memory 
  shared_data_t* sdata = (shared_data_t*) shmat(shmid,(void*)0,0); 
  sdata-&gt;value = 0;
  </code></pre>
  
  <p><strong>Reader:</strong></p>
  
  <pre><code>// ftok to generate unique key 
  key_t key = ftok("shmfile",65); 
  
  // shmget returns an identifier in shmid 
  int shmid = shmget(key,1024,0666|IPC_CREAT); 
  ftruncate(shmid, sizeof(shared_data_t));
  
  // shmat to attach to shared memory 
  shared_data_t* sdata = (shared_data_t*) shmat(shmid,(void*)0,0); 
  </code></pre>
  
  <p><strong>Initialization of Synchronization Tools in Writer</strong></p>
  
  <pre><code>pthread_mutexattr_t mutex_attr;
  pthread_mutexattr_init(&amp;mutex_attr);
  pthread_mutexattr_setpshared(&amp;mutex_attr, PTHREAD_PROCESS_SHARED);
  pthread_mutex_init(&amp;sdata-&gt;ipc_mutex, &amp;mutex_attr);
  sem_init(&amp;sdata-&gt;ipc_sem, 1, 0);
  </code></pre>
  
  <p><strong>Write Code</strong></p>
  
  <pre><code>for (int i = 0; i &lt; 20000000; ++i)
      {
          pthread_mutex_lock(&amp;sdata-&gt;ipc_mutex);
          sdata-&gt;value++;
          clock_gettime(CLOCK_MONOTONIC, &amp;sdata-&gt;ts);
          sdata-&gt;start_time = (BILLION*sdata-&gt;ts.tv_sec) + sdata-&gt;ts.tv_nsec;
          sdata-&gt;read_cond = 1;
          pthread_mutex_unlock(&amp;sdata-&gt;ipc_mutex);
          sem_wait(&amp;sdata-&gt;ipc_sem);
      }
  fprintf(stderr, "done writing\n" );
  
  pthread_mutex_lock(&amp;sdata-&gt;ipc_mutex);
  sdata-&gt;end_cond = 1;
  pthread_mutex_unlock(&amp;sdata-&gt;ipc_mutex);
  </code></pre>
  
  <p><strong>Read Code</strong> </p>
  
  <pre><code>double counter = 0;
  double total_time = 0;
  double max_time = 0;
  double min_time = BILLION;
  double max_thresh = 1000;
  int above_max_counter = 0;
  double last_val = 0;
  while (1) {
  
          pthread_mutex_lock(&amp;sdata-&gt;ipc_mutex);
          while (!sdata-&gt;read_cond &amp;&amp; !sdata-&gt;end_cond) {
              pthread_mutex_unlock(&amp;sdata-&gt;ipc_mutex);
              pthread_mutex_lock(&amp;sdata-&gt;ipc_mutex);
          }
  
          clock_gettime(CLOCK_MONOTONIC, &amp;sdata-&gt;ts);
          double time_to_read = (BILLION*sdata-&gt;ts.tv_sec) + sdata-&gt;ts.tv_nsec - sdata-&gt;start_time;
  
          if (sdata-&gt;end_cond) {
              break;
          }
  
          if (sdata-&gt;value != last_val + 1) {
              fprintf(stderr, "synchronization error: val: %g, last val: %g\n", sdata-&gt;value, last_val);
          }
          last_val = sdata-&gt;value;
  
          if (time_to_read &gt; max_time) {
              max_time = time_to_read;
              printf("max time: %lf, counter: %ld\n", max_time, (long int) counter);
          }  
          if (time_to_read &lt; min_time) min_time = time_to_read;
  
          if (time_to_read &gt; max_thresh) above_max_counter++;
          total_time += time_to_read;
          counter++;
  
          sdata-&gt;read_cond = 0;
          sem_post(&amp;sdata-&gt;ipc_sem);
          pthread_mutex_unlock(&amp;sdata-&gt;ipc_mutex);
  
      }
  
  fprintf(stderr, "avg time to read: %g\n", total_time / counter);
  fprintf(stderr, "max time to read: %g\n", max_time);
  fprintf(stderr, "min time to read: %g\n", min_time);
  fprintf(stderr, "count above max threshhold of %g ns: %d\n", max_thresh, above_max_counter);
  </code></pre>
  
  <p><strong>Cleanup in Writer</strong></p>
  
  <pre><code>//detach from shared memory 
  shmdt(sdata); 
  </code></pre>
  
  <p><strong>Cleanup in Reader</strong></p>
  
  <pre><code>pthread_mutex_unlock(&amp;sdata-&gt;ipc_mutex);
  pthread_mutex_destroy(&amp;sdata-&gt;ipc_mutex);
  
  //detach from shared memory 
  shmdt(sdata); 
  
  // destroy the shared memory 
  shmctl(shmid,IPC_RMID,NULL); 
  </code></pre>
  
  <p>The goal is to minimize the amount of time spent between these two operations. Ideally, I would like to be able to guarantee that the time to read from the time the value is written is less than 1 microsecond. However, the output I get:</p>
  
  <pre><code>max time: 5852.000000, counter: 0
  max time: 18769.000000, counter: 30839
  max time: 27416.000000, counter: 66632
  max time: 28668.000000, counter: 1820109
  max time: 121362.000000, counter: 1853346
  done writing
  avg time to read: 277.959
  max time to read: 121362
  min time to read: 60
  count above max threshhold of 1000 ns: 1871
  </code></pre>
  
  <p>indicates that there are a number of times (~.01% of reads) where the read exceeds 1 us, and can go as high as 121us.</p>
  
  <p>My question is as follows:</p>
  
  <p><strong>What could be causing these spikes, since I have set the priority to highest and isolated the CPU on which these programs are running?</strong></p>
  
  <p>I have learned from <a href="https://stackoverflow.com/questions/16275444/how-to-print-time-difference-in-accuracy-of-milliseconds-and-nanoseconds-from-c">this post</a> that I should not expect clock_gettime to have nanosecond accuracy. Are these spikes simply inaccuracies in clock_gettime?</p>
  
  <p>The other option I considered is that these cores (6 and 7) are being interrupted somehow, despite having been set as highest priority.</p>
  
  <p>Any help would be greatly appreciated.</p>
  
  <p><strong>EDIT</strong></p>
  
  <p>Per comment below, here is the contents of my <code>/proc/interrupts</code> file:</p>
  
  <pre><code>           CPU0       CPU1       CPU2       CPU3       CPU4       CPU5       CPU6       CPU7       
     0:         20          0          0          0          0          0          0          0   IO-APIC    2-edge      timer
     1:          2          0          0          0          0          0          0          0   IO-APIC    1-edge      i8042
     8:          1          0          0          0          0          0          0          0   IO-APIC    8-edge      rtc0
     9:          0          0          0          0          0          0          0          0   IO-APIC    9-fasteoi   acpi
    12:          2          0          0          0          1          1          0          0   IO-APIC   12-edge      i8042
    16:          0          0          0          0          0          0          0          0   IO-APIC   16-fasteoi   i801_smbus, pcim_das1602_16
    19:          2          0          0          0          8         10          6          2   IO-APIC   19-fasteoi 
   120:          0          0          0          0          0          0          0          0   PCI-MSI 16384-edge      aerdrv
   121:         99        406          0          0         14       5960          6          0   PCI-MSI 327680-edge      xhci_hcd
   122:       8726        133         47         28       4126       3910      22638        795   PCI-MSI 376832-edge      ahci[0000:00:17.0]
   123:          2          0          0          0          2          0          3       3663   PCI-MSI 520192-edge      eno1
   124:       3411          0          2          1        176      24498         77         11   PCI-MSI 32768-edge      i915
   125:         45          0          0          0          3          6          0          0   PCI-MSI 360448-edge      mei_me
   126:        432          0          0          0        144        913         28          1   PCI-MSI 514048-edge      snd_hda_intel:card0
   NMI:          1          1          1          1          1          1          1          1   Non-maskable interrupts
   LOC:      12702      10338      10247      10515       9969      10386      16658      13568   Local timer interrupts
   SPU:          0          0          0          0          0          0          0          0   Spurious interrupts
   PMI:          1          1          1          1          1          1          1          1   Performance monitoring interrupts
   IWI:          0          0          0          0          0          0          0          0   IRQ work interrupts
   RTR:          7          0          0          0          0          0          0          0   APIC ICR read retries
   RES:       4060       2253       1026        708        595        846        887        751   Rescheduling interrupts
   CAL:      11906      10423      11418       9894      14562      11000      21479      11223   Function call interrupts
   TLB:      10620       8996      10060       8674      13172       9622      20121       9838   TLB shootdowns
   TRM:          0          0          0          0          0          0          0          0   Thermal event interrupts
   THR:          0          0          0          0          0          0          0          0   Threshold APIC interrupts
   DFR:          0          0          0          0          0          0          0          0   Deferred Error APIC interrupts
   MCE:          0          0          0          0          0          0          0          0   Machine check exceptions
   MCP:          2          2          2          2          2          2          2          2   Machine check polls
   ERR:          0
   MIS:          0
   PIN:          0          0          0          0          0          0          0          0   Posted-interrupt notification event
   PIW:          0          0          0          0          0          0          0          0   Posted-interrupt wakeup event
  </code></pre>
  
  <p>I've tried changing the smp affinity for interrupts 122 and 123 to cores 0 and 1, per <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/performance_tuning_guide/s-cpu-irq" rel="nofollow noreferrer">this post</a>, which appears to do nothing, as when I reset my computer, these affinities are still set to cores 6 and 7, respectively.</p>
  
  <p>Even without resetting and simply re-running my programs(s), I see no change in the number of interrupts serviced by these CPU cores.</p>