title: How to understand "/proc/[pid]/stack"?
body: <p>According to <code>proc</code> manual:  </p>
  
  <blockquote>
    <p>/proc/[pid]/stack (since Linux 2.6.29)  </p>
    
    <p>This file provides a symbolic trace of the function calls in
                  this process's kernel stack.  This file is provided only if
                  the kernel was built with the CONFIG_STACKTRACE configuration
                  option.  </p>
  </blockquote>
  
  <p>So I write a program to test:  </p>
  
  <pre class="lang-c prettyprint-override"><code>#include &lt;stdio.h&gt;
  #include &lt;sys/wait.h&gt;
  #include &lt;unistd.h&gt;
  #include &lt;pthread.h&gt;
  
  void *thread_func(void *p_arg)
  {
          pid_t pid = fork();
          if (pid &gt; 0) {
              wait(NULL);
              return 0;
          } else if (pid == 0) {
              sleep(1000);
              return 0;
          }
          return NULL;
  }
  int main(void)
  {
          pthread_t t1, t2;
  
          pthread_create(&amp;t1, NULL, thread_func, "Thread 1");
          pthread_create(&amp;t2, NULL, thread_func, "Thread 2");
  
          sleep(1000);
          return 0;
  }
  </code></pre>
  
  <p>After running, use <code>pstack</code> to check the threads of progress:  </p>
  
  <pre><code>linux-uibj:~ # pstack 24976
  Thread 3 (Thread 0x7fd6e4ed5700 (LWP 24977)):
  #0  0x00007fd6e528d3f4 in wait () from /lib64/libpthread.so.0
  #1  0x0000000000400744 in thread_func ()
  #2  0x00007fd6e52860a4 in start_thread () from /lib64/libpthread.so.0
  #3  0x00007fd6e4fbb7fd in clone () from /lib64/libc.so.6
  Thread 2 (Thread 0x7fd6e46d4700 (LWP 24978)):
  #0  0x00007fd6e528d3f4 in wait () from /lib64/libpthread.so.0
  #1  0x0000000000400744 in thread_func ()
  #2  0x00007fd6e52860a4 in start_thread () from /lib64/libpthread.so.0
  #3  0x00007fd6e4fbb7fd in clone () from /lib64/libc.so.6
  Thread 1 (Thread 0x7fd6e569f700 (LWP 24976)):
  #0  0x00007fd6e4f8d6cd in nanosleep () from /lib64/libc.so.6
  #1  0x00007fd6e4f8d564 in sleep () from /lib64/libc.so.6
  #2  0x00000000004007b1 in main ()
  </code></pre>
  
  <p>At the same time, check <code>/proc/24976/stack</code>:  </p>
  
  <pre><code>linux-uibj:~ # cat /proc/24976/stack
  [&lt;ffffffff804ba1a7&gt;] system_call_fastpath+0x16/0x1b
  [&lt;00007fd6e4f8d6cd&gt;] 0x7fd6e4f8d6cd
  [&lt;ffffffffffffffff&gt;] 0xffffffffffffffff
  </code></pre>
  
  <p>The <code>24976</code> process has <code>3</code> threads, and they all block on system call(<code>nanosleep</code> and <code>wait</code>), so all <code>3</code> threads now work in <code>kernel</code> space, and turn into <strong>kernel threads</strong> now, right? If this is true, there should be <code>3</code> stacks in <code>/proc/[pid]/stack</code> file. But it seems there is only <code>1</code> stack in <code>/proc/[pid]/stack</code> file.</p>
  
  <p>How should I understand <code>/proc/[pid]/stack</code>?</p>