# syz-testbed

`syz-testbed` is the tool that simplifies the process of evaluating the
performance of different syzkaller versions (or configurations) against each
other. The tool automates checking out syzkaller repos, building them, running
`syz-manager`s and collecting/summarizing their results.

## Configuring syz-testbed

`syz-testbed` requires a JSON config file. An example of such a file:

```json
{
  "workdir": "/tmp/syz-testbed-workdir/",
  "corpus": "/tmp/corpus.db",
  "target": "syz-manager",
  "max_instances": 5,
  "run_time": "24h",
  "http": "0.0.0.0:50000",
  "checkouts": [
    {
      "name": "first",
      "repo": "https://github.com/google/syzkaller.git",
    },
    {
      "name": "second",
      "repo": "https://github.com/google/syzkaller.git",
      "branch": "some-dev-branch",
    }
  ],
  "manager_config": {
	  "target": "linux/amd64",
	  "kernel_obj": "/tmp/linux-stable",
	  "image": "/tmp/kernel-image/bullseye.img",
	  "sshkey": "/tmp/kernel-image/bullseye.id_rsa",
	  "procs": 8,
	  "type": "qemu",
	  "vm": {
          "count": 2,
          "kernel": "/tmp/linux-stable/arch/x86/boot/bzImage",
          "cpu": 2,
          "mem": 2048
	  }
  }
}
```

Given such a configuration file, `syz-testbed` will do the following:
1. Check out the `master` branch of `https://github.com/google/syzkaller.git`
   into `/tmp/syz-testbed-workdir/checkouts/first/` and build it.
2. Check out the `some-dev-branch` of `https://github.com/google/syzkaller.git`
   into `/tmp/syz-testbed-workdir/checkouts/second/` and build it.
3. Set up and run 3 instances of `first` and 2 instances of `second`
(`max_instances = 5`).

The directory structure looks as follows:
```
/tmp/syz-testbed-workdir/
└── checkouts
    ├── first
    │   ├── run-first-0
    │   │   ├── log.txt
    │   │   ├── manager.cfg
    │   │   └── workdir
    │   ├── run-first-1
    │   │   ├── log.txt
    │   │   ├── manager.cfg
    │   │   └── workdir
    │   └── run-first-4
    │   │   ├── log.txt
    │   │   ├── manager.cfg
    │   │   └── workdir
    └── second
        ├── run-second-2
        │   ├── log.txt
        │   ├── manager.cfg
        │   └── workdir
        └── run-second-3
            ├── log.txt
            ├── manager.cfg
            └── workdir
```
4. After 24 hours (as `run_hours` is 24), stop those 5 instances.
5. Create and run 2 instances of `first` and 3 instances of `second`.
6. <Repeat those steps over and over>

The tool stops after receiving a SIGINT (e.g. after Ctrl+C) or a SIGTERM
signal. Also, if one of the instances has exited due to some error, this also
stops the whole experiment.

## Web interface

The tool has a simple web interface that displays the current information about
the experiment (the number of active and finished instances, the time until
instances are stopped, etc.) and the latest statistics collected from the
`syz-manager`s.

If the `benchmp` parameter points to the `syz-benchcmp` executable, then the web
interface can also generate graphs of various parameters over time or the number
of executions.

In order to enable the interface, set the `http` parameter to the IP address and
port to which `syz-testbed` should bind. E.g. `"http": "0.0.0.0:50000"`.

## Statistics

`syz-testbed` provides two "views" of the statistics:
1. `complete` - only includes data from the finished instances (i.e. those that
   have been running for `run_hours`).
2. `all` - also includes the data from the currently active instances. The
   statistics from the finished instances is winded back to match the current
   uptime of the active instances.

Therefore, the statistics is laid out the following way.

```bash
$ tree -L 2 /tmp/syz-testbed-workdir/
/tmp/syz-testbed-workdir/
├── stats_all
│   ├── benches
│   │   ├── avg_first.txt
│   │   ├── avg_second.txt
│   ├── bugs.csv
│   ├── checkout_stats.csv
│   └── instance_stats.csv
├── stats_completed
│   ├── benches
│   │   ├── avg_first.txt
│   │   ├── avg_second.txt
│   ├── bugs.csv
│   ├── checkout_stats.csv
│   └── instance_stats.csv
└── testbed.csv
```

1. `bugs.csv` contains all the bugs found by the running instances. If a single
   checkout has several instances (i.e. `count` > 1), `syz-testbed` takes a
   union of bugs found by them. The purpose is ultimately to collect all bugs
   that could be found by that version of syzkaller.
2. Statistics that is generated by individual `syz-manager`s is saved into
   `instance_stats.csv`. The same data is also averaged among instances that
   belong to the same checkouts and saved into the `checkout_stats.csv` file.
3. Bench files (see `tools/syz-benchcmp`) of all `syz-manager`s belonging to a
   single checkout are averaged and saved into the corresponding files in the
   `benches` folder.

The statics is updated once every 90 seconds.

## Running syz-testbed

First, checkout the most recent version of syzkaller itself:

```
$ git clone https://github.com/google/syzkaller.git
```

Then, build `syz-testbed`:

```
$ cd syzkaller/tools/syz-testbed/
$ go build
```

Write and save the configuration file (e.g. into the `config.json` file). Then,
`syz-testbed` can be run using the following command:

```
$ ./syz-testbed -config config.json
```

Stopping the `syz-testbed` process results in stopping all the syzkaller instances.

## Testing syz-repro

`syz-testbed` can also be used to test syzkaller's ability to reproduce bugs. To do
so, set the `target` property in the `syz-testbed`'s config file to `syz-repro`.

One can also specify the source of the crash log files. This is either just a folder,
whose files will be treated accordingly or it can be a syzkaller's workdir.
`input_logs` must point to the folde with crash logs - `syz-testbed` will traverse
it and treat each file as an input. Otherwise, `input_workdir` must be used.

For example:
```json
  "repro_config": {
    "input_workdir": "/tmp/some-syzkaller-workdir",
    "crashes_per_bug": 2,
    "skip_bugs": ["SYZFAIL", "no output", "corrupted", "lost connection"]
  },
```

In this case, `syz-testbed` will traverse all bugs found by the syzkaller, skip
those that match "SYZFAIL", "no output", "corrupted" or "lost connection", then
pick 2 random crash logs for each such bug for later processing.

`syz-testbed` will check out and compile the specified syzkaller instances and will
go on executing their `syz-repro`s on each picked up crash log file, as long as
the tool is not stopped.
