Ftrace

ftrace (Function Tracer) is a tracing framework for the Linux kernel. Although its original name, Function Tracer, came from ftrace's ability to record information related to various function calls performed while the kernel is running, ftrace's tracing capabilities cover a much broader range of kernel's internal operations.
Details.
With its various tracer plugins, ftrace can be targeted at different static tracepoints, such as scheduling events, interrupts, memory-mapped I/O, CPU power state transitions, and operations related to file systems and virtualization. Also, dynamic tracking of kernel function calls is available, optionally restrictable to a subset of functions by using globs, and with the possibility to generate call graphs and provide stack usage reports. At the same time, ftrace can be used to measure various latencies within the Linux kernel, such as for how long interrupts or preemption are disabled.
An ftrace-enabled Linux kernel is built by enabling the kernel configuration option. The entire runtime interaction with ftrace is performed through readable and writable virtual files contained in a specifically mounted debugfs file system; as a result, ftrace requires no specialized userspace utilities to operate. However, there are additional userspace utilities that provide more advanced features for data recording, analysis and visualization; examples of such utilities are "trace-cmd" and "KernelShark".
Internally, ftrace relies on the gcc's profiling mechanism to prepend machine instructions to the compiled versions of all source-level kernel functions, which redirect the execution of functions to the ftrace's trampolines and tracer plugins that perform the actual tracing. These "entry point" instructions created by gcc are altered by ftrace when the kernel is booted, and varied later at runtime by ftrace between NOPs and actual jumps to the tracing trampolines, depending on the tracing types and options configured at runtime.
ftrace is developed primarily by Steven Rostedt, and it was merged into the Linux kernel mainline in kernel version 2.6.27, which was released on October 9, 2008.