title: How to find out which kernel spinlock eat up most of CPU?
body: <p>I'm doing performance tuning of a crypto software, which is run on Linux and utilizes hardware crypto acceleration device.</p>
  
  <p>When the load is given over some threshold, kernel <code>_spn_lock</code> begin to eat most of the CPU's time. The following <code>perf top</code> screenshot shows ~30% of CPU is taken by <code>_spin_lock</code>, but it goes up over 50% if a load is increased and overall performance degrades.</p>
  
  <p>I can not tune up the performance of the software without finding out which spinlock causes this situation. </p>
  
  <p>I checked up similar questions on stackoverflow, but I cannot find a answer on <strong>how to discover which spinlock spins</strong>.
  Is there any way to locate the spinlock position? Any tips or suggestions would be deeply appreciated.</p>
  
  <pre><code>   PerfTop:      81 irqs/sec  kernel:45.7%  exact:  0.0% [1000Hz cycles],  (all, CPU: 53)
  ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
  
               samples  pcnt function                DSO
               _______ _____ _______________________ __________________________
  
                 44.00 27.2% _spin_lock              [kernel.kallsyms]
                 30.00 18.5% sha512_block_data_order /usr/local/bin/cryptod
                  6.00  3.7% _int_malloc             /lib64/libc-2.12.so
                  5.00  3.1% __GI___libc_malloc      /lib64/libc-2.12.so
  
  
  
  [root@qat ~]# uname -a
  Linux qat 2.6.32-220.el6.tsx.65.x86_64 #68 SMP Fri Jan 26 17:45:32 KST 2018 x86_64 x86_64 x86_64 GNU/Linux
  </code></pre>