title: Need help in reading callgrind output
body: <p>I have run callgrind with my application like this:</p>
  
  <pre><code>valgrind --tool=callgrind MyApplication
  </code></pre>
  
  <p>and then call:</p>
  
  <pre><code>callgrind_annotate --auto=yes ./callgrind.out.2489
  </code></pre>
  
  <p>I see output like:</p>
  
  <pre><code> 768,097,560  PROGRAM TOTALS
  
  --------------------------------------------------------------------------------
         Ir  file:function
  --------------------------------------------------------------------------------
  18,624,794  /build/buildd/eglibc-2.11.1/elf/dl-lookup.c:do_lookup_x
  [/lib/ld-2.11.1.so]
  18,149,492  /src/js/src/jsgc.cpp:JS_CallTracer'2
  [/src/firefox-debug-objdir/js/src/libmozjs.so]
  16,328,897 /src/layout/style/nsCSSDataBlock.cpp:nsCSSExpandedDataBlock::DoAssertInitialState()
  [/src/firefox-debug-objdir/toolkit/library/libxul.so]
  13,376,634  /build/buildd/eglibc-2.11.1/nptl/pthread_getspecific.c:pthread_getspecific
  [/lib/libpthread-2.11.1.so]
  13,005,623  /build/buildd/eglibc-2.11.1/malloc/malloc.c:_int_malloc
  [/lib/libc-2.11.1.so]
  10,404,453  ???:0x0000000000009190 [/usr/lib/libpangocairo-1.0.so.0.2800.0]
  10,358,646  /src/xpcom/io/nsFastLoadFile.cpp:NS_AccumulateFastLoadChecksum(unsigned
  int*, unsigned char const*, unsigned int, int)
  [/src/firefox-debug-objdir/toolkit/library/libxul.so]
   8,543,634  /src/js/src/jsscan.cpp:js_GetToken
  [/src/firefox-debug-objdir/js/src/libmozjs.so]
   7,451,273  /src/xpcom/typelib/xpt/src/xpt_arena.c:XPT_ArenaMalloc
  [/src/firefox-debug-objdir/toolkit/library/libxul.so]
   7,335,131  ???:g_type_check_instance_is_a [/usr/lib/libgobject-2.0.so.0.2400.0]
  </code></pre>
  
  <p>I have a few questions:</p>
  
  <ol>
  <li><p>What does the number on the right mean? Does it mean it spend accumulative that long in calling the function on the right? How can I tell how many times that function has been called and Does that include the time spend in calling the functions called by that function?</p></li>
  <li><p>What does line with <code>???</code> mean? e.g. <code>???:0x0000000000009190 [/usr/lib/libpangocairo-1.0.so.0.2800.0]</code></p></li>
  </ol>