title: Is it normal that the gcc atomic builtins are so slow?
body: <p>I have an application where I have to increment some statistics counters in a multi-threaded method. The incrementing has to be thread-safe, so I decided to use the gcc atomic builtins <code>__sync_add_and_fetch()</code> function. Just to get an idea of their impact, I did some simple performance testing and noticed that these functions are much slower than simple pre/post incrementing.</p>
  
  <p>Here is the test program that I created:</p>
  
  <pre><code>#include &lt;iostream&gt;
  #include &lt;pthread.h&gt;
  #include &lt;time.h&gt;
  
  using namespace std;
  
  uint64_t diffTimes(struct timespec &amp;start, struct timespec &amp;end)
  {
    if(start.tv_sec == end.tv_sec)
    {
      return end.tv_nsec - start.tv_nsec;
    }
    else if(start.tv_sec &lt; end.tv_sec)
    {
      uint64_t nsecs = (end.tv_sec - start.tv_sec) * 1000000000;
      return nsecs + end.tv_nsec - start.tv_nsec;
    }
    else
    {
      // this is actually an error
      return 0;
    }
  }
  
  void outputResult(const char *msg, struct timespec &amp;start, struct timespec &amp;end, uint32_t numIterations, uint64_t val)
  {
    uint64_t diff = diffTimes(start, end);
    cout &lt;&lt; msg &lt;&lt; ": "
         &lt;&lt; "\n\t iterations: " &lt;&lt; numIterations
         &lt;&lt; ", result: " &lt;&lt; val
         &lt;&lt; "\n\t times [start, end] =  [" &lt;&lt; start.tv_sec &lt;&lt; ", " &lt;&lt; start.tv_nsec &lt;&lt; "]"
         &lt;&lt; "\n\t [" &lt;&lt; end.tv_sec &lt;&lt; ", " &lt;&lt; end.tv_nsec &lt;&lt; "]"
         &lt;&lt; "\n\t [total, avg] = [" &lt;&lt; diff
         &lt;&lt; ", " &lt;&lt; (diff/numIterations) &lt;&lt; "] nano seconds"
         &lt;&lt; endl;
  }
  
  int main(int argc, char **argv)
  {
    struct timespec start, end;
    uint64_t val = 0;
    uint32_t numIterations = 1000000;
  
    //
    // NON ATOMIC pre increment
    //
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;start);
    for(uint32_t i = 0; i &lt; numIterations; ++i)
    {
      ++val;
    }
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;end);
  
    outputResult("Non-Atomic pre-increment", start, end, numIterations, val);
    val = 0;
  
    //
    // NON ATOMIC post increment
    //
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;start);
    for(uint32_t i = 0; i &lt; numIterations; ++i)
    {
      val++;
    }
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;end);
  
    outputResult("Non-Atomic post-increment", start, end, numIterations, val);
    val = 0;
  
    //
    // ATOMIC add and fetch
    //
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;start);
    for(uint32_t i = 0; i &lt; numIterations; ++i)
    {
      __sync_add_and_fetch(&amp;val, 1);
    }
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;end);
  
    outputResult("Atomic add and fetch", start, end, numIterations, val);
    val = 0;
  
    //
    // ATOMIC fetch and add
    //
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;start);
    for(uint32_t i = 0; i &lt; numIterations; ++i)
    {
      __sync_fetch_and_add(&amp;val, 1);
    }
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;end);
  
    outputResult("Atomic fetch and add", start, end, numIterations, val);
    val = 0;
  
    //
    // Mutex protected post-increment
    //
    pthread_mutex_t mutex;
    pthread_mutex_init(&amp;mutex, NULL);
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;start);
    for(uint32_t i = 0; i &lt; numIterations; ++i)
    {
      pthread_mutex_lock(&amp;mutex);
      val++;
      pthread_mutex_unlock(&amp;mutex);
    }
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;end);
  
    outputResult("Mutex post-increment", start, end, numIterations, val);
    val = 0;
  
    //
    // RWlock protected post-increment
    //
    pthread_rwlock_t rwlock;
    pthread_rwlock_init(&amp;rwlock, NULL);
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;start);
    for(uint32_t i = 0; i &lt; numIterations; ++i)
    {
      pthread_rwlock_wrlock(&amp;rwlock);
      val++;
      pthread_rwlock_unlock(&amp;rwlock);
    }
    clock_gettime(CLOCK_PROCESS_CPUTIME_ID, &amp;end);
  
    outputResult("RWlock post-increment", start, end, numIterations, val);
    val = 0;
  
    return 0;
  }
  </code></pre>
  
  <p>And here are the results:</p>
  
  <pre><code># ./atomicVsNonAtomic
  Non-Atomic pre-increment:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 1585375]
           [0, 1586185]
           [total, avg] = [810, 0] nano seconds
  Non-Atomic post-increment:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 1667489]
           [0, 1667920]
           [total, avg] = [431, 0] nano seconds
  Atomic add and fetch:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 1682037]
           [0, 16595016]
           [total, avg] = [14912979, 14] nano seconds
  Atomic fetch and add:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 16617178]
           [0, 31499571]
           [total, avg] = [14882393, 14] nano seconds
  Mutex post-increment:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 31526810]
           [0, 68515763]
           [total, avg] = [36988953, 36] nano seconds
  RWlock post-increment:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 68547649]
           [0, 110877351]
           [total, avg] = [42329702, 42] nano seconds
  </code></pre>
  
  <p>Here is the gcc compilation:</p>
  
  <pre><code>g++ -o atomicVsNonAtomic.o -c -march=i686 -O2 -I. atomicVsNonAtomic.cc
  g++ -o atomicVsNonAtomic atomicVsNonAtomic.o -lrt -lpthread
  </code></pre>
  
  <p>And related info and versions:</p>
  
  <pre><code># gcc --version
  gcc (GCC) 4.3.2
  Copyright (C) 2008 Free Software Foundation, Inc.
  This is free software; see the source for copying conditions.  There is NO
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
  
  # uname -a
  Linux gtcba2v1 2.6.32.12-0.7-default #1 SMP 2010-05-20 11:14:20 +0200 i686 i686 i386 GNU/Linux
  </code></pre>
  
  <p>And now for the actual question :) Is it normal that the atomic operations are so much slower? </p>
  
  <p>The difference for one million iterations is:</p>
  
  <ul>
  <li>simple post-increment: 431 nano seconds</li>
  <li>atomic fetch and add operation: 14882393 nano seconds</li>
  </ul>
  
  <p>Of course I understand that an atomic operation should be more costly, but this seems exaggerated. Just for completeness, I also checked a pthread mutex and rwlock. At least the atomic operations are faster then the pthread operations, but Im still wondering if I've done something wrong. I couldnt get it to link without specifying the <code>-march=i686</code> option, maybe this has an impact?</p>
  
  <p><em><strong>UPDATE:</em></strong></p>
  
  <p>I took out the <code>-O2</code> compiler optimization and was able to get more coherent results as follows: </p>
  
  <pre><code># ./atomicVsNonAtomic
  Non-Atomic pre-increment:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 1647303]
           [0, 4171164]
           [total, avg] = [2523861, 2] nano seconds
  Non-Atomic post-increment:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 4310230]
           [0, 7262704]
           [total, avg] = [2952474, 2] nano seconds
  Atomic add and fetch:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 7285996]
           [0, 25919067]
           [total, avg] = [18633071, 18] nano seconds
  Atomic fetch and add:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 25941677]
           [0, 44544234]
           [total, avg] = [18602557, 18] nano seconds
  Mutex post-increment:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 44573933]
           [0, 82318615]
           [total, avg] = [37744682, 37] nano seconds
  RWlock post-increment:
           iterations: 1000000, result: 1000000
           times [start, end] =  [0, 82344866]
           [0, 125124498]
           [total, avg] = [42779632, 42] nano seconds
  </code></pre>