在 Fedora 中用 bpftrace 追踪代码
bpftrace
![](/data/attachment/album/202107/29/222633wjln78my68exj7x9.jpg)


bpftrace 是一个 [基于 eBPF 的新型追踪工具](https://github.com/iovisor/bpftrace)，在 Fedora 28 第一次引入。Brendan Gregg、Alastair Robertson 和 Matheus Marchini 在网上的一个松散的黑客团队的帮助下开发了 bpftrace。它是一个允许你分析系统在幕后正在执行的操作的追踪工具，可以告诉你代码中正在被调用的函数、传递给函数的参数、函数的调用次数等。


这篇文章的内容涉及了 bpftrace 的一些基础，以及它是如何工作的，请继续阅读获取更多的信息和一些有用的实例。


### eBPF（<ruby> 扩展的伯克利数据包过滤器 <rt>  extended Berkeley Packet Filter </rt></ruby>）


[eBPF](https://lwn.net/Articles/740157/) 是一个微型虚拟机，更确切的说是一个位于 Linux 内核中的虚拟 CPU。eBPF 可以在内核空间以一种安全可控的方式加载和运行小型程序，使得 eBPF 的使用更加安全，即使在生产环境系统中。eBPF 虚拟机有自己的指令集架构（[ISA](https://github.com/iovisor/bpf-docs/blob/master/eBPF.md)），类似于现代处理器架构的一个子集。通过这个 ISA，可以很容易将 eBPF 程序转化为真实硬件上的代码。内核即时将程序转化为主流处理器架构上的本地代码，从而提升性能。


eBPF 虚拟机允许通过编程扩展内核，目前已经有一些内核子系统使用这一新型强大的 Linux 内核功能，比如网络、安全计算、追踪等。这些子系统的主要思想是添加 eBPF 程序到特定的代码点，从而扩展原生的内核行为。


虽然 eBPF 机器语言功能强大，由于是一种底层语言，直接用于编写代码很费力，bpftrace 就是为了解决这个问题而生的。eBPF 提供了一种编写 eBPF 追踪脚本的高级语言，然后在 clang / LLVM 库的帮助下将这些脚本转化为 eBPF，最终添加到特定的代码点。


### 安装和快速入门


在终端 [使用](https://fedoramagazine.org/howto-use-sudo/) [sudo](https://fedoramagazine.org/howto-use-sudo/) 执行下面的命令安装 bpftrace：



```
$ sudo dnf install bpftrace

```

使用“hello world”进行实验：



```
$ sudo bpftrace -e 'BEGIN { printf("hello world\n"); }'

```

注意，出于特权级的需要，你必须使用 root 运行 `bpftrace`，使用 `-e` 选项指明一个程序，构建一个所谓的“单行程序”。这个例子只会打印 “hello world”，接着等待你按下 `Ctrl+C`。


`BEGIN` 是一个特殊的探针名，只在执行一开始生效一次；每次探针命中时，大括号 `{}` 内的操作（这个例子中只是一个 `printf`）都会执行。


现在让我们转向一个更有用的例子：



```
$ sudo bpftrace -e 't:syscalls:sys_enter_execve { printf("%s called %s\n", comm, str(args->filename)); }'

```

这个例子打印了父进程的名字（`comm`）和系统中正在创建的每个新进程的名称。`t:syscalls:sys_enter_execve` 是一个内核追踪点，是 `tracepoint:syscalls:sys_enter_execve` 的简写，两种形式都可以使用。下一部分会向你展示如何列出所有可用的追踪点。


`comm` 是一个 bpftrace 内建指令，代表进程名；`filename` 是 `t:syscalls:sys_enter_execve` 追踪点的一个字段，这些字段可以通过 `args` 内建指令访问。


追踪点的所有可用字段可以通过这个命令列出：



```
bpftrace -lv "t:syscalls:sys_enter_execve"

```

### 示例用法


#### 列出探针


`bpftrace` 的一个核心概念是<ruby> 探针点 <rt>  probe point </rt></ruby>，即 eBPF 程序可以连接到的（内核或用户空间的）代码中的测量点，可以分成以下几大类：


* `kprobe`——内核函数的开始处
* `kretprobe`——内核函数的返回处
* `uprobe`——用户级函数的开始处
* `uretprobe`——用户级函数的返回处
* `tracepoint`——内核静态追踪点
* `usdt`——用户级静态追踪点
* `profile`——基于时间的采样
* `interval`——基于时间的输出
* `software`——内核软件事件
* `hardware`——处理器级事件


所有可用的 `kprobe` / `kretprobe`、`tracepoints`、`software` 和 `hardware` 探针可以通过这个命令列出：



```
$ sudo bpftrace -l

```

`uprobe` / `uretprobe` 和 `usdt` 是用户空间探针，专用于某个可执行文件。要使用这些探针，通过下文中的特殊语法。


`profile` 和 `interval` 探针以固定的时间间隔触发；固定的时间间隔不在本文的范畴内。


#### 统计系统调用数


**映射** 是保存计数、统计数据和柱状图的特殊 BPF 数据类型，你可以使用映射统计每个系统调用正在被调用的次数：



```
$ sudo bpftrace -e 't:syscalls:sys_enter_* { @[probe] = count(); }'

```

一些探针类型允许使用通配符匹配多个探针，你也可以使用一个逗号隔开的列表为一个操作块指明多个连接点。上面的例子中，操作块连接到了所有名称以 `t:syscalls:sysenter_` 开头的追踪点，即所有可用的系统调用。


`bpftrace` 的内建函数 `count()` 统计系统调用被调用的次数；`@[]` 代表一个映射（一个关联数组）。该映射的键 `probe` 是另一个内建指令，代表完整的探针名。


这个例子中，相同的操作块连接到了每个系统调用，之后每次有系统调用被调用时，映射就会被更新，映射中和系统调用对应的项就会增加。程序终止时，自动打印出所有声明的映射。


下面的例子统计所有的系统调用，然后通过 `bpftrace` 过滤语法使用 PID 过滤出某个特定进程调用的系统调用：



```
$ sudo bpftrace -e 't:syscalls:sys_enter_* / pid == 1234 / { @[probe] = count(); }'

```

#### 进程写的字节数


让我们使用上面的概念分析每个进程正在写的字节数：



```
$ sudo bpftrace -e 't:syscalls:sys_exit_write /args->ret > 0/ { @[comm] = sum(args->ret); }'

```

`bpftrace` 连接操作块到写系统调用的返回探针（`t:syscalls:sys_exit_write`），然后使用过滤器丢掉代表错误代码的负值（`/arg->ret > 0/`）。


映射的键 `comm` 代表调用系统调用的进程名；内建函数 `sum()` 累计每个映射项或进程写的字节数；`args` 是一个 `bpftrace` 内建指令，用于访问追踪点的参数和返回值。如果执行成功，`write` 系统调用返回写的字节数，`arg->ret` 用于访问这个字节数。


#### 进程的读取大小分布（柱状图）：


`bpftrace` 支持创建柱状图。让我们分析一个创建进程的 `read` 大小分布的柱状图的例子：



```
$ sudo bpftrace -e 't:syscalls:sys_exit_read { @[comm] = hist(args->ret); }'

```

柱状图是 BPF 映射，因此必须保存为一个映射（`@`），这个例子中映射键是 `comm`。


这个例子使 `bpftrace` 给每个调用 `read` 系统调用的进程生成一个柱状图。要生成一个全局柱状图，直接保存 `hist()` 函数到 `@`（不使用任何键）。


程序终止时，`bpftrace` 自动打印出声明的柱状图。创建柱状图的基准值是通过 *args->ret* 获取到的读取的字节数。


#### 追踪用户空间程序


你也可以通过 `uprobes` / `uretprobes` 和 USDT（用户级静态定义的追踪）追踪用户空间程序。下一个例子使用探测用户级函数结尾处的 `uretprobe` ，获取系统中运行的每个 `bash` 发出的命令行：



```
$ sudo bpftrace -e 'uretprobe:/bin/bash:readline { printf("readline: \"%s\"\n", str(retval)); }'

```

要列出可执行文件 `bash` 的所有可用 `uprobes` / `uretprobes`， 执行这个命令：



```
$ sudo bpftrace -l "uprobe:/bin/bash"

```

`uprobe` 指向用户级函数执行的开始，`uretprobe` 指向执行的结束（返回处）；`readline()` 是 `/bin/bash` 的一个函数，返回键入的命令行；`retval` 是被探测的指令的返回值，只能在 `uretprobe` 访问。


使用 `uprobes` 时，你可以用 `arg0..argN` 访问参数。需要调用 `str()` 将 `char *` 指针转化成一个字符串。


### 自带脚本


`bpftrace` 软件包附带了许多有用的脚本，可以在 `/usr/share/bpftrace/tools/` 目录找到。


这些脚本中，你可以找到：


* `killsnoop.bt`——追踪 `kill()` 系统调用发出的信号
* `tcpconnect.bt`——追踪所有的 TCP 网络连接
* `pidpersec.bt`——统计每秒钟（通过 `fork`）创建的新进程
* `opensnoop.bt`——追踪 `open()` 系统调用
* `bfsstat.bt`——追踪一些 VFS 调用，按秒统计


你可以直接使用这些脚本，比如：



```
$ sudo /usr/share/bpftrace/tools/killsnoop.bt

```

你也可以在创建新的工具时参考这些脚本。


### 链接


* bpftrace 参考指南——<https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md>
* Linux 2018 `bpftrace`（DTrace 2.0）——<http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html>
* BPF：通用的内核虚拟机——<https://lwn.net/Articles/599755/>
* Linux Extended BPF（eBPF）Tracing Tools——<http://www.brendangregg.com/ebpf.html>
* 深入 BPF：一个阅读材料列表—— [https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf](https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/)




---


via: <https://fedoramagazine.org/trace-code-in-fedora-with-bpftrace/>


作者：[Augusto Caringi](https://fedoramagazine.org/author/acaringi/) 选题：[lujun9972](https://github.com/lujun9972) 译者：[YungeG](https://github.com/YungeG) 校对：[wxy](https://github.com/wxy)


本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出


OK
bpftrace is [a new eBPF-based tracing tool](https://github.com/iovisor/bpftrace) that was first included in Fedora 28. It was developed by Brendan Gregg, Alastair Robertson and Matheus Marchini with the help of a loosely-knit team of hackers across the Net. A tracing tool lets you analyze what a system is doing behind the curtain. It tells you which functions in code are being called, with which arguments, how many times, and so on.

This article covers some basics about bpftrace, and how it works. Read on for more information and some useful examples.

#### eBPF (extended Berkeley Packet Filter)

[eBPF](https://lwn.net/Articles/740157/) is a tiny virtual machine, or a virtual CPU to be more precise, in the Linux Kernel. The eBPF can load and run small programs in a safe and controlled way in kernel space. This makes it safer to use, even in production systems. This virtual machine has its own instruction set architecture ([ISA](https://github.com/iovisor/bpf-docs/blob/master/eBPF.md)) resembling a subset of modern processor architectures. The ISA makes it easy to translate those programs to the real hardware. The kernel performs just-in-time translation to native code for main architectures to improve the performance.

The eBPF virtual machine allows the kernel to be extended programmatically. Nowadays several kernel subsystems take advantage of this new powerful Linux Kernel capability. Examples include networking, seccomp, tracing, and more. The main idea is to attach eBPF programs into specific code points, and thereby extend the original kernel behavior.

eBPF machine language is very powerful. But writing code directly in it is extremely painful, because it’s a low level language. This is where bpftrace comes in. It provides a high-level language to write eBPF tracing scripts. The tool then translates these scripts to eBPF with the help of clang/LLVM libraries, and then attached to the specified code points.

## Installation and quick start

To install bpftrace, run the following command in a terminal [using ](https://fedoramagazine.org/howto-use-sudo/)* sudo*:

$ sudo dnf install bpftrace

Try it out with a “hello world” example:

$ sudo bpftrace -e 'BEGIN { printf("hello world\n"); }'

Note that you must run *bpftrace* as *root* due to the privileges required. Use the *-e* option to specify a program, and to construct the so-called “one-liners.” This example only prints *hello world*, and then waits for you to press **Ctrl+C**.

*BEGIN* is a special probe name that fires only once at the beginning of execution. Every action inside the curly braces *{ }* fires whenever the probe is hit — in this case, it’s just a *printf*.

Let’s jump now to a more useful example:

$ sudo bpftrace -e 't:syscalls:sys_enter_execve { printf("%s called %s\n", comm, str(args->filename)); }'

This example prints the parent process name *(comm)* and the name of every new process being created in the system. *t:syscalls:sys_enter_execve* is a kernel tracepoint. It’s a shorthand for *tracepoint:syscalls:sys_enter_execve*, but both forms can be used. The next section shows you how to list all available tracepoints.

*comm* is a bpftrace builtin that represents the process name. *filename* is a field of the *t:syscalls:sys_enter_execve* tracepoint. You can access these fields through the *args* builtin.

All available fields of the tracepoint can be listed with this command:

bpftrace -lv "t:syscalls:sys_enter_execve"

## Example usage

### Listing probes

A central concept for *bpftrace* are **probe points**. Probe points are instrumentation points in code (kernel or userspace) where eBPF programs can be attached. They fit into the following categories:

*kprobe*– kernel function start*kretprobe*– kernel function return*uprobe*– user-level function start*uretprobe*– user-level function return*tracepoint*– kernel static tracepoints*usdt*– user-level static tracepoints*profile*– timed sampling*interval*– timed output*software*– kernel software events*hardware*– processor-level events

All available *kprobe/kretprobe*, *tracepoints*, *software* and *hardware* probes can be listed with this command:

$ sudo bpftrace -l

The *uprobe/uretprobe* and *usdt* probes are userspace probes specific to a given executable. To use them, use the special syntax shown later in this article.

The *profile* and *interval* probes fire at fixed time intervals. Fixed time intervals are not covered in this article.

### Counting system calls

**Maps** are special BPF data types that store counts, statistics, and histograms. You can use maps to summarize how many times each syscall is being called:

$ sudo bpftrace -e 't:syscalls:sys_enter_* { @[probe] = count(); }'

Some probe types allow wildcards to match multiple probes. You can also specify multiple attach points for an action block using a comma separated list. In this example, the action block attaches to all tracepoints whose name starts with *t:syscalls:sys_enter_*, which means all available syscalls.

The bpftrace builtin function *count()* counts the number of times this function is called. *@[]* represents a map (an associative array). The key of this map is *probe*, which is another bpftrace builtin that represents the full probe name.

Here, the same action block is attached to every syscall. Then, each time a syscall is called the map will be updated, and the entry is incremented in the map relative to this same syscall. When the program terminates, it automatically prints out all declared maps.

This example counts the syscalls called globally, it’s also possible to filter for a specific process by *PID* using the bpftrace filter syntax:

$ sudo bpftrace -e 't:syscalls:sys_enter_* / pid == 1234 / { @[probe] = count(); }'

### Write bytes by process

Using these concepts, let’s analyze how many bytes each process is writing:

$ sudo bpftrace -e 't:syscalls:sys_exit_write /args->ret > 0/ { @[comm] = sum(args->ret); }'

*bpftrace* attaches the action block to the write syscall return probe (*t:syscalls:sys_exit_write*). Then, it uses a filter to discard the negative values, which are error codes *(/args->ret > 0/)*.

The map key *comm* represents the process name that called the syscall. The *sum()* builtin function accumulates the number of bytes written for each map entry or process. *args* is a bpftrace builtin to access tracepoint’s arguments and return values. Finally, if successful, the *write* syscall returns the number of written bytes. *args->ret* provides access to the bytes.

### Read size distribution by process (histogram):

*bpftrace* supports the creation of histograms. Let’s analyze one example that creates a histogram of the *read* size distribution by process:

$ sudo bpftrace -e 't:syscalls:sys_exit_read { @[comm] = hist(args->ret); }'

Histograms are BPF maps, so they must always be attributed to a map (*@*). In this example, the map key is *comm*.

The example makes *bpftrace* generate one histogram for every process that calls the *read* syscall. To generate just one global histogram, attribute the *hist()* function just to *‘@’* (without any key).

bpftrace automatically prints out declared histograms when the program terminates. The value used as base for the histogram creation is the number of read bytes, found through *args->ret*.

### Tracing userspace programs

You can also trace userspace programs with *uprobes/uretprobes* and *USDT* (User-level Statically Defined Tracing). The next example uses a *uretprobe*, which probes to the end of a user-level function. It gets the command lines issued in every *bash* running in the system:

$ sudo bpftrace -e 'uretprobe:/bin/bash:readline { printf("readline: \"%s\"\n", str(retval)); }'

To list all available *uprobes/uretprobes* of the *bash* executable, run this command:

$ sudo bpftrace -l "uprobe:/bin/bash"

*uprobe* instruments the beginning of a user-level function’s execution, and *uretprobe* instruments the end (its return). *readline()* is a function of */bin/bash*, and it returns the typed command line. *retval* is the return value for the instrumented function, and can only be accessed on *uretprobe*.

When using *uprobes*, you can access arguments with *arg0..argN*. A *str()* call is necessary to turn the *char ** pointer to a *string*.

## Shipped Scripts

There are many useful scripts shipped with bpftrace package. You can find them in the */usr/share/bpftrace/tools/* directory.

Among them, you can find:

*killsnoop.bt*– Trace signals issued by the kill() syscall.*tcpconnect.bt*– Trace all TCP network connections.*pidpersec.bt*– Count new procesess (via fork) per second.*opensnoop.bt*– Trace open() syscalls.*vfsstat.bt*– Count some VFS calls, with per-second summaries.

You can directly use the scripts. For example:

$ sudo /usr/share/bpftrace/tools/killsnoop.bt

You can also study these scripts as you create new tools.

## Links

- bpftrace reference guide –
[https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md](https://github.com/iovisor/bpftrace/blob/master/docs/reference_guide.md) - bpftrace (DTrace 2.0) for Linux 2018 –
[http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html](http://www.brendangregg.com/blog/2018-10-08/dtrace-for-linux-2018.html) - BPF: the universal in-kernel virtual machine –
[https://lwn.net/Articles/599755/](https://lwn.net/Articles/599755/) - Linux Extended BPF (eBPF) Tracing Tools –
[http://www.brendangregg.com/ebpf.html](http://www.brendangregg.com/ebpf.html) - Dive into BPF: a list of reading material –
[https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf](https://qmonnet.github.io/whirl-offload/2016/09/01/dive-into-bpf/)

*Photo by **Roman Romashov** on **Unsplash**.*

## child

Fantastic! And is it available in Centos 7?

## Konstantin

Great article! Wonder if bpftrace can be used inside bash scripts.

## Peter van Gemert

Yes it is possible to use bpftrace in shell scripts. bpftrace is a cli command that can be run from the prompt or from shell scripts.