title: What does perf's option to measure events at user and kernel levels mean?
body: <p>The Linux perf tool provides access to CPU event counters. It lets you specify the events to be counted and when to count those events.</p>
  
  <p><a href="https://perf.wiki.kernel.org/index.php/Tutorial" rel="noreferrer">https://perf.wiki.kernel.org/index.php/Tutorial</a></p>
  
  <blockquote>
    <p>By default, events are measured at both user and kernel levels:</p>
    
    <p><code>perf stat -e cycles dd if=/dev/zero of=/dev/null count=100000</code></p>
    
    <p>To measure only at the user level, it is necessary to pass a modifier:</p>
    
    <p><code>perf stat -e cycles:u dd if=/dev/zero of=/dev/null count=100000</code></p>
    
    <p>To measure both user and kernel (explicitly):</p>
    
    <p><code>perf stat -e cycles:uk dd if=/dev/zero of=/dev/null count=100000</code></p>
  </blockquote>
  
  <p>From this, I expected that <code>cycles:u</code> meant "only count events while running non-kernel code" and recorded counts would not map to kernel symbols but that doesn't seem to be the case.</p>
  
  <p>Here's an example:</p>
  
  <pre><code>perf record -e cycles:u du -sh ~
  [...]
  perf report --stdio -i perf.data
  [...]
  9.24%  du       [kernel.kallsyms]  [k] system_call
  [...]
  0.70%  du       [kernel.kallsyms]  [k] page_fault
  [...]
  </code></pre>
  
  <p>If I do the same but use <code>cycles:uk</code> then I do get more kernel symbols reported so the event modifiers do have an effect. Using <code>cycles:k</code> produces reports with almost exclusively kernel symbols but it does include a few libc symbols.</p>
  
  <p>What's going on here? Is this the expected behavior? Am I misunderstanding the language used in the linked document?</p>
  
  <p>The linked document also includes this table which uses slightly different descriptions if that helps:</p>
  
  <blockquote>
  <pre><code>Modifiers | Description                          | Example
  ----------+--------------------------------------+----------
  u         | monitor at priv level 3, 2, 1 (user) | event:u
  k         | monitor at priv level 0 (kernel)     | event:k
  </code></pre>
  </blockquote>
  
  <p><strong>Edit:</strong> more info:</p>
  
  <p>CPU is an Intel Haswell. The specific model is an i7-5820K.</p>
  
  <p>Distro is up to date Arch Linux (rolling release schedule) with kernel 4.1.6.</p>
  
  <p>The version of perf itself is 4.2.0.</p>
  
  <p><strong>Edit2:</strong></p>
  
  <p>More output from example runs. As you can see, <code>cycles:u</code> mostly reports non-kernel symbols. I know that perf sometimes mis-attributes counts to a neighboring instruction when you look at the annotated assembly output. Maybe this is related?</p>
  
  <p>cycles:u</p>
  
  <pre><code># perf record -e cycles:u du -sh ~
  179G    /home/khouli
  [ perf record: Woken up 1 times to write data ]
  [ perf record: Captured and wrote 0.116 MB perf.data (2755 samples) ]
  # sudo perf report --stdio -i perf.data
  # To display the perf.data header info, please use --header/--header-only options.
  #
  #
  # Total Lost Samples: 0
  #
  # Samples: 2K of event 'cycles:u'
  # Event count (approx.): 661835375
  #
  # Overhead  Command  Shared Object      Symbol
  # ........  .......  .................  ..............................
  #
      11.02%  du       libc-2.22.so       [.] _int_malloc
       9.73%  du       libc-2.22.so       [.] _int_free
       9.24%  du       du                 [.] fts_read
       9.23%  du       [kernel.kallsyms]  [k] system_call
       4.17%  du       libc-2.22.so       [.] strlen
       4.17%  du       libc-2.22.so       [.] __memmove_sse2
       3.47%  du       libc-2.22.so       [.] __readdir64
       3.33%  du       libc-2.22.so       [.] malloc_consolidate
       2.87%  du       libc-2.22.so       [.] malloc
       1.83%  du       libc-2.22.so       [.] msort_with_tmp.part.0
       1.63%  du       libc-2.22.so       [.] __memcpy_avx_unaligned
       1.63%  du       libc-2.22.so       [.] __getdents64
       1.52%  du       libc-2.22.so       [.] free
       1.47%  du       libc-2.22.so       [.] __memmove_avx_unaligned
       1.44%  du       du                 [.] 0x000000000000e609
       1.41%  du       libc-2.22.so       [.] _wordcopy_bwd_dest_aligned
       1.19%  du       du                 [.] 0x000000000000e644
       0.93%  du       libc-2.22.so       [.] __fxstatat64
       0.85%  du       libc-2.22.so       [.] do_fcntl
       0.73%  du       [kernel.kallsyms]  [k] page_fault
  [lots more symbols, almost all in du...]
  </code></pre>
  
  <p>cycles:uk</p>
  
  <pre><code># perf record -e cycles:uk du -sh ~
  179G    /home/khouli
  [ perf record: Woken up 1 times to write data ]
  [ext4] with build id 0f47443e26a238299e8a5963737da23dd3530376 not found,
  continuing without symbols
  [ perf record: Captured and wrote 0.120 MB perf.data (2856 samples) ]
  # perf report --stdio -i perf.data
  # To display the perf.data header info, please use --header/--header-only options.
  #
  #
  # Total Lost Samples: 0
  #
  # Samples: 2K of event 'cycles:uk'
  # Event count (approx.): 3118065867
  #
  # Overhead  Command  Shared Object      Symbol
  # ........  .......  .................  ..............................................
  #
      13.80%  du       [kernel.kallsyms]  [k] __d_lookup_rcu
       6.16%  du       [kernel.kallsyms]  [k] security_inode_getattr
       2.52%  du       [kernel.kallsyms]  [k] str2hashbuf_signed
       2.43%  du       [kernel.kallsyms]  [k] system_call
       2.35%  du       [kernel.kallsyms]  [k] half_md4_transform
       2.31%  du       [kernel.kallsyms]  [k] ext4_htree_store_dirent
       1.97%  du       [kernel.kallsyms]  [k] copy_user_enhanced_fast_string
       1.96%  du       libc-2.22.so       [.] _int_malloc
       1.93%  du       du                 [.] fts_read
       1.90%  du       [kernel.kallsyms]  [k] system_call_after_swapgs
       1.83%  du       libc-2.22.so       [.] _int_free
       1.44%  du       [kernel.kallsyms]  [k] link_path_walk
       1.33%  du       libc-2.22.so       [.] __memmove_sse2
       1.19%  du       [kernel.kallsyms]  [k] _raw_spin_lock
       1.19%  du       [kernel.kallsyms]  [k] __fget_light
       1.12%  du       [kernel.kallsyms]  [k] kmem_cache_alloc
       1.12%  du       [kernel.kallsyms]  [k] __ext4_check_dir_entry
       1.05%  du       [kernel.kallsyms]  [k] lockref_get_not_dead
       1.02%  du       [kernel.kallsyms]  [k] generic_fillattr
       0.95%  du       [kernel.kallsyms]  [k] do_dentry_open
       0.95%  du       [kernel.kallsyms]  [k] path_init
       0.95%  du       [kernel.kallsyms]  [k] lockref_put_return
       0.91%  du       libc-2.22.so       [.] do_fcntl
       0.91%  du       [kernel.kallsyms]  [k] ext4_getattr
       0.91%  du       [kernel.kallsyms]  [k] rb_insert_color
       0.88%  du       [kernel.kallsyms]  [k] __kmalloc
       0.88%  du       libc-2.22.so       [.] __readdir64
       0.88%  du       libc-2.22.so       [.] malloc
       0.84%  du       [kernel.kallsyms]  [k] ext4fs_dirhash
       0.84%  du       [kernel.kallsyms]  [k] __slab_free
       0.84%  du       [kernel.kallsyms]  [k] in_group_p
       0.81%  du       [kernel.kallsyms]  [k] get_empty_filp
       0.77%  du       libc-2.22.so       [.] malloc_consolidate
  [more...]
  </code></pre>
  
  <p>cycles:k</p>
  
  <pre><code># perf record -e cycles:k du -sh ~
  179G    /home/khouli
  [ perf record: Woken up 1 times to write data ]
  [ext4] with build id 0f47443e26a238299e8a5963737da23dd3530376 not found, continuing
  without symbols
  [ perf record: Captured and wrote 0.118 MB perf.data (2816 samples) ]
  # perf report --stdio -i perf.data
  # To display the perf.data header info, please use --header/--header-only options.
  #
  #
  # Total Lost Samples: 0
  #
  # Samples: 2K of event 'cycles:k'
  # Event count (approx.): 2438426748
  #
  # Overhead  Command  Shared Object      Symbol
  # ........  .......  .................  ..............................................
  #
      17.11%  du       [kernel.kallsyms]  [k] __d_lookup_rcu
       6.97%  du       [kernel.kallsyms]  [k] security_inode_getattr
       4.22%  du       [kernel.kallsyms]  [k] half_md4_transform
       3.10%  du       [kernel.kallsyms]  [k] str2hashbuf_signed
       3.01%  du       [kernel.kallsyms]  [k] system_call_after_swapgs
       2.59%  du       [kernel.kallsyms]  [k] ext4_htree_store_dirent
       2.24%  du       [kernel.kallsyms]  [k] copy_user_enhanced_fast_string
       2.14%  du       [kernel.kallsyms]  [k] lockref_get_not_dead
       1.86%  du       [kernel.kallsyms]  [k] ext4_getattr
       1.85%  du       [kernel.kallsyms]  [k] kfree
       1.68%  du       [kernel.kallsyms]  [k] __ext4_check_dir_entry
       1.53%  du       [kernel.kallsyms]  [k] __fget_light
       1.34%  du       [kernel.kallsyms]  [k] link_path_walk
       1.34%  du       [kernel.kallsyms]  [k] path_init
       1.22%  du       [kernel.kallsyms]  [k] __kmalloc
       1.22%  du       [kernel.kallsyms]  [k] kmem_cache_alloc
       1.14%  du       [kernel.kallsyms]  [k] do_dentry_open
       1.11%  du       [kernel.kallsyms]  [k] ext4_readdir
       1.07%  du       [kernel.kallsyms]  [k] __find_get_block_slow
       1.07%  du       libc-2.22.so       [.] do_fcntl
       1.04%  du       [kernel.kallsyms]  [k] _raw_spin_lock
       0.99%  du       [kernel.kallsyms]  [k] _raw_read_lock
       0.95%  du       libc-2.22.so       [.] __fxstatat64
       0.94%  du       [kernel.kallsyms]  [k] rb_insert_color
       0.94%  du       [kernel.kallsyms]  [k] generic_fillattr
       0.93%  du       [kernel.kallsyms]  [k] ext4fs_dirhash
       0.93%  du       [kernel.kallsyms]  [k] find_get_entry
       0.89%  du       [kernel.kallsyms]  [k] rb_next
       0.89%  du       [kernel.kallsyms]  [k] is_dx_dir
       0.89%  du       [kernel.kallsyms]  [k] in_group_p
       0.89%  du       [kernel.kallsyms]  [k] cp_new_stat
     [more...]
  </code></pre>
  
  <p>perf_event_paranoid</p>
  
  <pre><code>$ cat /proc/sys/kernel/perf_event_paranoid
  1
  </code></pre>
  
  <p>kernel config for perf</p>
  
  <pre><code>$ cat /proc/config.gz | gunzip | grep -A70 'Kernel Perf'
  # Kernel Performance Events And Counters
  #
  CONFIG_PERF_EVENTS=y
  # CONFIG_DEBUG_PERF_USE_VMALLOC is not set
  CONFIG_VM_EVENT_COUNTERS=y
  CONFIG_SLUB_DEBUG=y
  # CONFIG_COMPAT_BRK is not set
  # CONFIG_SLAB is not set
  CONFIG_SLUB=y
  CONFIG_SLUB_CPU_PARTIAL=y
  CONFIG_SYSTEM_TRUSTED_KEYRING=y
  CONFIG_PROFILING=y
  CONFIG_TRACEPOINTS=y
  CONFIG_OPROFILE=m
  # CONFIG_OPROFILE_EVENT_MULTIPLEX is not set
  CONFIG_HAVE_OPROFILE=y
  CONFIG_OPROFILE_NMI_TIMER=y
  CONFIG_KPROBES=y
  CONFIG_JUMP_LABEL=y
  CONFIG_KPROBES_ON_FTRACE=y
  CONFIG_UPROBES=y
  # CONFIG_HAVE_64BIT_ALIGNED_ACCESS is not set
  CONFIG_HAVE_EFFICIENT_UNALIGNED_ACCESS=y
  CONFIG_ARCH_USE_BUILTIN_BSWAP=y
  CONFIG_KRETPROBES=y
  CONFIG_USER_RETURN_NOTIFIER=y
  CONFIG_HAVE_IOREMAP_PROT=y
  CONFIG_HAVE_KPROBES=y
  CONFIG_HAVE_KRETPROBES=y
  CONFIG_HAVE_OPTPROBES=y
  CONFIG_HAVE_KPROBES_ON_FTRACE=y
  CONFIG_HAVE_ARCH_TRACEHOOK=y
  CONFIG_HAVE_DMA_ATTRS=y
  CONFIG_HAVE_DMA_CONTIGUOUS=y
  CONFIG_GENERIC_SMP_IDLE_THREAD=y
  CONFIG_HAVE_REGS_AND_STACK_ACCESS_API=y
  CONFIG_HAVE_CLK=y
  CONFIG_HAVE_DMA_API_DEBUG=y
  CONFIG_HAVE_HW_BREAKPOINT=y
  CONFIG_HAVE_MIXED_BREAKPOINTS_REGS=y
  CONFIG_HAVE_USER_RETURN_NOTIFIER=y
  CONFIG_HAVE_PERF_EVENTS_NMI=y
  CONFIG_HAVE_PERF_REGS=y
  CONFIG_HAVE_PERF_USER_STACK_DUMP=y
  CONFIG_HAVE_ARCH_JUMP_LABEL=y
  CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG=y
  CONFIG_HAVE_ALIGNED_STRUCT_PAGE=y
  CONFIG_HAVE_CMPXCHG_LOCAL=y
  CONFIG_HAVE_CMPXCHG_DOUBLE=y
  CONFIG_ARCH_WANT_COMPAT_IPC_PARSE_VERSION=y
  CONFIG_ARCH_WANT_OLD_COMPAT_IPC=y
  CONFIG_HAVE_ARCH_SECCOMP_FILTER=y
  CONFIG_SECCOMP_FILTER=y
  CONFIG_HAVE_CC_STACKPROTECTOR=y
  CONFIG_CC_STACKPROTECTOR=y
  # CONFIG_CC_STACKPROTECTOR_NONE is not set
  # CONFIG_CC_STACKPROTECTOR_REGULAR is not set
  CONFIG_CC_STACKPROTECTOR_STRONG=y
  CONFIG_HAVE_CONTEXT_TRACKING=y
  CONFIG_HAVE_VIRT_CPU_ACCOUNTING_GEN=y
  CONFIG_HAVE_IRQ_TIME_ACCOUNTING=y
  CONFIG_HAVE_ARCH_TRANSPARENT_HUGEPAGE=y
  CONFIG_HAVE_ARCH_HUGE_VMAP=y
  CONFIG_HAVE_ARCH_SOFT_DIRTY=y
  CONFIG_MODULES_USE_ELF_RELA=y
  CONFIG_HAVE_IRQ_EXIT_ON_IRQ_STACK=y
  CONFIG_ARCH_HAS_ELF_RANDOMIZE=y
  CONFIG_OLD_SIGSUSPEND3=y
  CONFIG_COMPAT_OLD_SIGACTION=y
  </code></pre>