title: Capturing user-space assembly with ftrace and kprobes (by using virtual address translation)?
body: <p>Apologies for the longish post, I'm having trouble formulating it in a shorter way. Also, maybe this is more appropriate for Unix &amp; Linux Stack Exchange, but I'll try here at SO first, as there is an <code>ftrace</code> tag.</p>
  
  <p>Anyways - I'd like to observe do machine instructions of a user program execute in the context of a full <code>function_graph</code> capture using <code>ftrace</code>. One problem is that I need this for an older kernel:</p>
  
  <pre><code>$ uname -a
  Linux mypc 2.6.38-16-generic #67-Ubuntu SMP Thu Sep 6 18:00:43 UTC 2012 i686 i686 i386 GNU/Linux
  </code></pre>
  
  <p>... and in this edition, there is no <code>UPROBES</code> - which, as <a href="https://lwn.net/Articles/499190/" rel="nofollow noreferrer">Uprobes in 3.5 [LWN.net]</a> notes, should be able to do something like that. (<sub>As long as I don't have to patch the original kernel, I would be willing to try a kernel module built out of tree, as <a href="http://www.chunghwan.com/en/systems/user-space-probes-uprobes/" rel="nofollow noreferrer">User-Space Probes (Uprobes) [chunghwan.com]</a> seems to demonstrate; but as far as I can see from <a href="http://lwn.net/Articles/433327/" rel="nofollow noreferrer">0: Inode based uprobes [LWN.net]</a>, the 2.6 would probably need a full patch</sub>)</p>
  
  <p>However, on this version, there is a <code>/sys/kernel/debug/kprobes</code>, and <code>/sys/kernel/debug/tracing/kprobe_events</code>; and <a href="http://lxr.free-electrons.com/source/Documentation/trace/kprobetrace.txt?v=2.6.38" rel="nofollow noreferrer">Documentation/trace/kprobetrace.txt</a> implies that a kprobe can be set directly on an address; even if I cannot find an example anywhere on how this is used.</p>
  
  <p>In any case, I would still not be sure what addresses to use - as a small example, let's say I want to trace the start of the <code>main</code> function of the <code>wtest.c</code> program (included below). I can do this to compile and obtain an machine instruction assembly listing:</p>
  
  <pre><code>$ gcc -g -O0 wtest.c -o wtest
  $ objdump -S wtest | less
  ...
  08048474 &lt;main&gt;:
  int main(void) {
   8048474:       55                      push   %ebp
   8048475:       89 e5                   mov    %esp,%ebp
   8048477:       83 e4 f0                and    $0xfffffff0,%esp
   804847a:       83 ec 30                sub    $0x30,%esp
   804847d:       65 a1 14 00 00 00       mov    %gs:0x14,%eax
   8048483:       89 44 24 2c             mov    %eax,0x2c(%esp)
   8048487:       31 c0                   xor    %eax,%eax
    char filename[] = "/tmp/wtest.txt";
  ...
    return 0;
   804850a:       b8 00 00 00 00          mov    $0x0,%eax
  }
  ...
  </code></pre>
  
  <p>I would set up ftrace logging via this script:</p>
  
  <pre><code>sudo bash -c '
  KDBGPATH="/sys/kernel/debug/tracing"
  echo function_graph &gt; $KDBGPATH/current_tracer
  echo funcgraph-abstime &gt; $KDBGPATH/trace_options
  echo funcgraph-proc &gt; $KDBGPATH/trace_options
  echo 0 &gt; $KDBGPATH/tracing_on
  echo &gt; $KDBGPATH/trace
  echo 1 &gt; $KDBGPATH/tracing_on ; ./wtest ; echo 0 &gt; $KDBGPATH/tracing_on
  cat $KDBGPATH/trace &gt; wtest.ftrace
  '
  </code></pre>
  
  <p>You can see a portion of the (otherwise complex) resulting <code>ftrace</code> log in  <a href="https://unix.stackexchange.com/questions/110411/observing-a-hard-disk-write-in-kernel-space-with-drivers-modules">debugging - Observing a hard-disk write in kernel space (with drivers/modules) - Unix &amp; Linux Stack Exchange</a> (where I got the example from).</p>
  
  <p>Basically, I'd want a printout in this <code>ftrace</code> log, when the first instructions of <code>main</code> - say, the instructions at 0x8048474, 0x8048475, 0x8048477, 0x804847a, 0x804847d, 0x8048483 and 0x8048487 - are executed by (any) CPU. The problem is, as far as I can understand from <a href="http://duartes.org/gustavo/blog/post/anatomy-of-a-program-in-memory" rel="nofollow noreferrer">Anatomy of a Program in Memory : Gustavo Duarte</a>, these addresses are the <em>virtual</em> addresses, as seen from the perspective of the process itself (and I gather, the same perspective is shown by <code>/proc/PID/maps</code>)... And apparently, for <code>krpobe_event</code> I'd need a physical address?</p>
  
  <p>So, my idea would be: if I can find the physical addresses corresponding to the virtual addresses of the program disassembly (say by coding a kernel module, which would accept pid and address, and return the physical address via procfs), I could set up addresses as a sort of "tracepoints" via <code>/sys/kernel/debug/tracing/kprobe_events</code> in the above script - and hopefully get them in the <code>ftrace</code> log. Could this work, in principle?</p>
  
  <p>One problem with this, I found on <a href="https://stackoverflow.com/questions/6245850/linuxubuntu-c-language-virtual-to-physical-address-translation">Linux(ubuntu), C language: Virtual to Physical Address Translation - Stack Overflow</a>:</p>
  
  <blockquote>
    <p>In user code, you can't know the physical address corresponding to a virtual address. This is information is simply not exported outside the kernel. It could even change at any time, especially if the kernel decides to swap out part of your process's memory.<br>
    ...<br>
    Pass the virtual address to the kernel using systemcall/procfs and use vmalloc_to_pfn. Return the Physical address through procfs/registers.</p>
  </blockquote>
  
  <p>However, <code>vmalloc_to_pfn</code> doesn't seem to be trivial either:</p>
  
  <p><a href="https://stackoverflow.com/questions/11442773/vmalloc-to-pfn-returns-32-bit-address-on-linux-32-system-why-does-it-chop-off-h">x86 64 - vmalloc_to_pfn returns 32 bit address on Linux 32 system. Why does it chop off higher bits of PAE physical address? - Stack Overflow</a></p>
  
  <blockquote>
    <p>VA: 0xf8ab87fc PA using vmalloc_to_pfn: 0x36f7f7fc. But I'm actually expecting: 0x136f7f7fc.<br>
    ...<br>
    The physical address falls between 4 to 5 GB. But I can't get the exact physical address, I only get the chopped off 32-bit address. Is there another way to get true physical address?</p>
  </blockquote>
  
  <p>So, I'm not sure how reliably I could extract the physical addresses so they are traced by kprobes - especially since "it could even change at any time". But here, I would hope that since the program is small and trivial, there would be a reasonable chance that the program would not swap while being traced, allowing for a proper capture to be obtained. (<sub>So even if I have to run the debug script above multiple times, as long as I can hope to obtain a "proper" capture once out of 10 times (or even 100 times), I'd be OK with it.</sub>).</p>
  
  <p>Note that I'd want an output through <code>ftrace</code>, so that the timestamps are expressed in the same domain (see <a href="https://stackoverflow.com/questions/20417897/reliable-linux-kernel-timestamps-or-adjustment-thereof-with-both-usbmon-and-ft">Reliable Linux kernel timestamps (or adjustment thereof) with both usbmon and ftrace? - Stack Overflow</a> for an illustration of a problem with timestamps). Thus, even if I could come up with, say, a <code>gdb</code> script, to run and trace the program from userspace (while simultaneously an <code>ftrace</code> capture is obtained) - I'd like to avoid that, as the overhead from <code>gdb</code> itself will show in the <code>ftrace</code> logs.</p>
  
  <p>So, in summary:</p>
  
  <ul>
  <li>Is the approach of obtaining (possibly through a separate kernel module) physical addresses from the virtual (from a disassembly of an executable) addresses - so they are used to trigger a kprobe_event logged by ftrace - worth pursuing? If so, are there any examples of kernel modules that can be used for this address translation purpose?</li>
  <li>Could I otherwise use a kernel module to "register" a callback/handler function when a particular memory address is being executed? Then I could simply use a <code>trace_printk</code> in that function to have an <code>ftrace</code> log (or even without that, the handler function name itself should show in the <code>ftrace</code> log), and it doesn't seem there will be too much overhead with that...</li>
  </ul>
  
  <p>Actually, in this 2007 posting, <a href="https://sourceware.org/ml/systemtap/2007-q1/msg00587.html" rel="nofollow noreferrer">Jim Keniston - utrace-based uprobes:  systemtap mailing list</a>, there is a <code>11. Uprobes Example</code> (added to <code>Documentation/uprobes.txt</code>), which seems to be exactly that - a kernel module registering a handler function. Unfortunately, it uses <code>linux/uprobes.h</code>; and I have only <code>kprobes.h</code> in my <code>/usr/src/linux-headers-2.6.38-16/include/linux/</code>. Also, on my system, even <code>systemtap</code> complains about <code>CONFIG_UTRACE</code> not being enabled (see <a href="https://stackoverflow.com/questions/311840/tool-to-trace-local-function-calls-in-linux#comment29656423_324709">this comment</a>)... So if there's any other approach I could use to obtain a debug trace like I want, without having to recompile the kernel to get uprobes, it would be great to know...</p>
  
  <hr>
  
  <p><code>wtest.c</code>:</p>
  
  <pre><code>#include &lt;stdio.h&gt;
  #include &lt;fcntl.h&gt;  // O_CREAT, O_WRONLY, S_IRUSR
  
  int main(void) {
    char filename[] = "/tmp/wtest.txt";
    char buffer[] = "abcd";
    int fd;
    mode_t perms = S_IRUSR|S_IWUSR|S_IRGRP|S_IWGRP|S_IROTH|S_IWOTH;
  
    fd = open(filename, O_RDWR|O_CREAT, perms);
    write(fd,buffer,4);
    close(fd);
  
    return 0;
  }
  </code></pre>