I'm trying to obtain a user stack trace in ftrace, but for some reason, only my user program cannot resolve the addresses to symbols. First, the OS I'm using is:$ uname -aLinux mypc 2.6.38-16-generic #67-Ubuntu SMP Thu Sep 6 18:00:43 UTC 2012 i686 i686 i386 GNU/LinuxThe (trivial) user-space program I'm using, wtest.c, is included below. I compile it with:gcc -g -pg -O0 wtest.c -o wtest... hoping that -g and -pg will include enough debugging symbol information.Then, I run the wtest program under ftrace. Note that the I want to obtain the user space stacktrace as a sort of a guide where in my user space program I am currently, while I otherwise run the function tracer. The Linux/Documentation/trace/ftrace.txt isn't explicit about this, but in order to have a stack trace, there must be something to trigger it - and just the function (or function_graph) tracer, on its own, will not do any such triggering. So I choose to trigger anytime a system call is detected via events/raw_syscalls. Furthermore, the stack trace will print addresses only, unless sym-userobj is specified. So, I run this script to do the tracing:sudo bash -c 'KDBGPATH=/sys/kernel/debug/tracingecho 1 > $KDBGPATH/events/raw_syscalls/enableecho userstacktrace > $KDBGPATH/trace_optionsecho sym-userobj > $KDBGPATH/trace_optionsecho function > $KDBGPATH/current_tracer ;echo 0 > $KDBGPATH/tracing_onecho > $KDBGPATH/traceecho 1 > $KDBGPATH/tracing_on ; /media/disk/wtest ; echo 0 > $KDBGPATH/tracing_oncat $KDBGPATH/trace > wtest.ftrace'However, the trace ends up containing something like this:...           wtest-16108 [000] 16246.413238: arch_flush_lazy_mmu_mode <-__kunmap_atomic            Xorg-1136  [001] 16246.413239: syscall_trace_leave <-syscall_exit_work           wtest-16108 [000] 16246.413240: up_read <-do_page_fault            Xorg-1136  [001] 16246.413241: sys_exit: NR 54 = 0            Xorg-1136  [001] 16246.413242: <user stack trace> =>  <008f9416> => /lib/i386-linux-gnu/libdrm_intel.so.1.0.0[+0x4f49] => /lib/i386-linux-gnu/libdrm_intel.so.1.0.0[+0x5cc3] => /lib/i386-linux-gnu/libdrm_intel.so.1.0.0[+0x13a3] => /usr/lib/xorg/modules/drivers/intel_drv.so[+0xec58] => /usr/lib/xorg/modules/drivers/intel_drv.so[+0x2957a] => /usr/lib/xorg/modules/drivers/intel_drv.so[+0x2d403] => /usr/bin/Xorg[+0xdefc8]            Xorg-1136  [001] 16246.413242: save_stack_trace_user <-ftrace_trace_userstack            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user            Xorg-1136  [001] 16246.413242: __copy_from_user_ll_nozero <-save_stack_trace_user           wtest-16108 [000] 16246.413242: syscall_trace_leave <-syscall_exit_work           wtest-16108 [000] 16246.413243: sys_exit: NR 120 = 0           wtest-16108 [000] 16246.413245: <user stack trace> =>  <00222416> =>  <08086a69> =>  <080753e0> =>  <080783b5> =>  <08073c09> =>  <08077c93> =>  <08078374> =>  <08073c09>           wtest-16108 [000] 16246.413245: save_stack_trace_user <-ftrace_trace_userstack           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user           wtest-16108 [000] 16246.413245: __copy_from_user_ll_nozero <-save_stack_trace_user           wtest-16108 [000] 16246.413260: down_read_trylock <-do_page_fault           wtest-16108 [000] 16246.413262: _cond_resched <-do_page_fault           wtest-16108 [000] 16246.413263: find_vma <-do_page_fault           wtest-16108 [000] 16246.413265: handle_mm_fault <-do_page_fault           wtest-16108 [000] 16246.413266: __kmap_atomic <-handle_mm_fault...In other words - Xorg has its user stack trace resolved fine (at least in the .so libraries) - and pretty much all other processes ending up in the ftrace log (like bash, etc); however, the user stack trace for wtest is printed with addresses only ?! I explicitly called the executable through its absolute path, but that doesn't seem to make much difference.Is there a way I could compile my wtest program, so its user stack trace gets its symbols resolved - or does the problem lie elsewhere? (I have tried -ggdb and -gdwarf-2 instead of -g, but those don't really help here...)Here is the code for wtest.c:#include <stdio.h>#include <fcntl.h>  // O_CREAT, O_WRONLY, S_IRUSRint main(void) {  char filename[] = /tmp/wtest.txt;  char buffer[] = abcd;  int fd;  mode_t perms = S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH|S_IWOTH;  fd = open(filename, O_RDWR|O_CREAT, perms);  write(fd,buffer,4);  close(fd);  return 0;}
Resolving executable user stack trace addresses in ftrace
linux kernel;debugging;trace