使用 parallel 利用起你的所有 CPU 资源
parallel CPU
![](/data/attachment/album/201712/22/223428y77ofxfxvxuufcuv.jpg)


bash 命令通常单线程运行。这意味着所有的处理工作只在单个 CPU 上执行。随着 CPU 规模的扩大以及核心数目的增加，这意味着只有一小部分的 CPU 资源用于处理你的工作。


当我们的工作受制于 CPU 处理数据的速度时，这些未使用的 CPU 资源能产生很大的效用。这种情况在进行多媒体转换（比如图片和视频转换）以及数据压缩中经常遇到。


本文中，我们将会使用 [parallel](https://www.gnu.org/software/parallel/) 程序。parallel 会接受一个列表作为输入，然后在所有 CPU 核上并行地执行命令来处理该列表。Parallel 甚至会按顺序将结果输出到标准输出中，因此它可以用在管道中作为其他命令的标准输入。


### 如何使用 parallel


parallel 在标准输入中读取一个列表作为输入，然后创建多个指定命令的进程来处理这个列表，其格式为：



```
list | parallel command

```

这里的 list 可以由任何常见的 bash 命令创建，例如：`cat`、`grep`、`find`。这些命令的结果通过管道从它们的标准输出传递到 parallel 的标准输入，像这样：



```
find . -type f -name "*.log" | parallel

```

跟 `find` 中使用 `-exec` 类似，`parallel` 使用 `{}` 来表示输入列表中的每个元素。下面这个例子中，`parallel` 会使用 `gzip` 压缩所有 `find` 命令输出的文件：



```
find . -type f -name "*.log" | parallel gzip {}

```

下面这些实际的使用 `parallel` 的例子可能会更容易理解一些。


### 使用 parallel 来进行 JPEG 压缩


在这个例子中，我收集了一些比较大的 `.jpg` 文件（大约 10MB 大小），要用 [Mozilla](https://www.mozilla.org/) 出品的 JPEG 图像压缩工具 [MozJPEG](https://github.com/mozilla/mozjpeg) 来进行处理。该工具会在尝试保持图像质量的同时减少 JPEG 图像文件的大小。这对降低网页加载时间很重要。


下面是一个普通的 `find` 命令，用来找出当前目录中的所有 `.jpg` 文件，然后通过 MozJPEG 包中提供的图像压缩工具 （`cjpeg`） 对其进行处理：



```
find . -type f -name "*.jpg" -exec cjpeg -outfile LoRes/{} {} ';'

```

总共耗时 `0m44.114s`。该命令运行时的 `top` 看起来是这样的：


![](/data/attachment/album/201712/22/223442u13nq87a61an9z9q.png)


你可以看到，虽然有 8 个核可用，但实际只有单个线程在用单个核。


下面用 `parallel` 来运行相同的命令：



```
find . -type f -name "*.jpg" | parallel cjpeg -outfile LoRes/{} {}

```

这次压缩所有图像的时间缩减到了 `0m10.814s`。从 `top` 显示中可以很清楚地看出不同：


![](/data/attachment/album/201712/22/223519h7n30u47omdqhzmu.png)


所有 CPU 核都满负荷运行，有 8 个线程对应使用 8 个 CPU 核。


### parallel 与 gzip 连用


如果你需要压缩多个文件而不是一个大文件，那么 `parallel` 就能用来提高处理速度。如果你需要压缩单个文件而同时又想要利用所有的 CPU 核的话，那么你应该 `gzip` 的多线程替代品 [pigz](https://zlib.net/pigz/)。


首先，我用随机数据创建了 100 个大约 1GB 的文件：



```
for i in {1..100}; do dd if=/dev/urandom of=file-$i bs=1MB count=10; done

```

然而我用 `find -exec` 命令来进行压缩：



```
find . -type f -name "file*" -exec gzip {} ';'

```

总共耗时 `0m28.028s`，而且也是只利用了单核。


换成 `parallel` 版本：



```
find . -type f -name "file*" | parallel gzip {}

```

耗时减少到了 `0m5.774s`。


parallel 是一款非常好用的工具，应该加入到你的系统管理工具包中，在合适的场合它能帮你节省大量的时间。




---


via: <https://bash-prompt.net/guides/parallell-bash/>


作者：[Elliot Cooper](https://bash-prompt.net/about) 译者：[lujun9972](https://github.com/lujun9972) 校对：[wxy](https://github.com/wxy)


本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出


OK
When a bash command is run it usually gets run in a single thread. This means that it will all the processing work will get executed on a single CPU. As CPU’s have scaled out and increased their core count this means that only a small fraction of the available CPU resources will get used to work on your process.

These unused CPU resources can make a big difference when the work we are trying to get done is bound by the speed that the CPU can crunch the data. This typically happens during media conversion e.g. picture and video, and data compression.

In this guide, we will look at using the bash program [Parallel](https://www.gnu.org/software/parallel/). Parallel works by accepting a list as input and then executing a command in parallel across all your CPU cores on that list. Parallel will even send any output to stdout in sequence so it can be piped as stdin for a further command.

**Update**: I’ve written an accompanying guide on naively using parallel to send jobs out to remote machines to use all of their cores [here](https://bash-prompt.net/guides/gnu-parallel-multi-server/).

## How To Use Parallel

Parallel takes a list as input on stdin and then creates a number of processes with a supplied command, this takes the form:

```
list | parallel command
```


The list can be created by any of the usual bash commands e.g. `cat`

, `grep`

, `find`

. The results of these commands are piped from their stdout to the stdin of parallel e.g.:

```
find . -type f -name "*.log" | parallel
```


Just like using `-exec`

with `find`

, `parallel`

substitutes each member in the input list as `{}`

. Here, `parallel`

will gzip every file that `find`

outputs:

```
find . -type f -name "*.log" | parallel gzip {}
```


The following examples of `parallel`

in action will make this easier to understand.

## Using Parallel For JPEG Optimization

In this example, I took a collection of largish `.jpg`

, ~10MB files and ran them through the [MozJPEG](https://github.com/mozilla/mozjpeg) JPEG image optimization tool produced by [Mozilla](https://www.mozilla.org/). This tool reduces JPEG image file size while attempting to retain the image quality. This is important for websites in order to keep page load times down.

Here is a typical `find`

command to locate every `.jpg`

file in the current directory and then run them through the image compression tool supplied in the MozJPEG package, `cjpeg`

:

```
find . -type f -name "*.jpg" -exec cjpeg -outfile LoRes/{} {} ';'
```


This took `0m44.114s`

seconds to run. Here is what `top`

looked like while it was running:

As you can see, only a single of the 8 available cores is working on the single thread.

Here is the same command run with `parallel`

:

```
find . -type f -name "*.jpg" | parallel cjpeg -outfile LoRes/{} {}
```


This reduces the time to optimize all the images to `0m10.814s`

. The difference is clearly seen in this image of `top`

:

All the CPU cores are maxed out and there are 8 threads to match the 8 available CPU cores.

## Using Parallel With GZIP

If you need to compress a number of files rather than a single large one then `parallel`

will speed things up. If you do need to compress a single file and want to utilize all your CPU cores take a look at the multi-threaded `gzip`

replacement [pigz](https://zlib.net/pigz/).

First, I created ~1GB of random data in 100 files:

```
for i in {1..100}; do dd if=/dev/urandom of=file-$i bs=1MB count=10; done
```


Then I compressed them using another `find -exec`

command:

```
find . -type f -name "file*" -exec gzip {} ';'
```


This took `0m28.028s`

to complete and again only used a single core.

Converting the same command to use `parallel`

gives us:

```
find . -type f -name "file*" | parallel gzip {}
```


This reduces the runtime to `0m5.774s`

.

Parallel is an easy to use tool that you should add to your sysadmins tool bag as it is going to save you a great deal of time in the right situation.