减少 curl 中内存分配操作（malloc）
curl malloc
![](/data/attachment/album/201709/16/115559adfoyopxdy6v95oc.jpg)


今天我在 libcurl 内部又做了[一个小改动](https://github.com/curl/curl/commit/cbae73e1dd95946597ea74ccb580c30f78e3fa73)，使其做更少的 malloc。这一次，泛型链表函数被转换成更少的 malloc (这才是链表函数应有的方式，真的)。


### 研究 malloc


几周前我开始研究内存分配。这很容易，因为多年前我们 curl 中就已经有内存调试和日志记录系统了。使用 curl 的调试版本，并在我的构建目录中运行此脚本：



```
#!/bin/sh
export CURL_MEMDEBUG=$HOME/tmp/curlmem.log
./src/curl http://localhost
./tests/memanalyze.pl -v $HOME/tmp/curlmem.log

```

对于 curl 7.53.1，这大约有 115 次内存分配。这算多还是少？


内存日志非常基础。为了让你有所了解，这是一个示例片段：



```
MEM getinfo.c:70 free((nil))
MEM getinfo.c:73 free((nil))
MEM url.c:294 free((nil))
MEM url.c:297 strdup(0x559e7150d616) (24) = 0x559e73760f98
MEM url.c:294 free((nil))
MEM url.c:297 strdup(0x559e7150d62e) (22) = 0x559e73760fc8
MEM multi.c:302 calloc(1,480) = 0x559e73760ff8
MEM hash.c:75 malloc(224) = 0x559e737611f8
MEM hash.c:75 malloc(29152) = 0x559e737a2bc8
MEM hash.c:75 malloc(3104) = 0x559e737a9dc8

```

### 检查日志


然后，我对日志进行了更深入的研究，我意识到在相同的代码行做了许多小内存分配。我们显然有一些相当愚蠢的代码模式，我们分配一个结构体，然后将该结构添加到链表或哈希，然后该代码随后再添加另一个小结构体，如此这般，而且经常在循环中执行。（我在这里说的是*我们*，不是为了责怪某个人，当然大部分的责任是我自己……）


这两种分配操作将总是成对地出现，并被同时释放。我决定解决这些问题。做非常小的（小于 32 字节）的分配也是浪费的，因为非常多的数据将被用于（在 malloc 系统内）跟踪那个微小的内存区域。更不用说堆碎片了。


因此，将该哈希和链表代码修复为不使用 malloc 是快速且简单的方法，对于最简单的 “curl http://localhost” 传输，它可以消除 20％ 以上的 malloc。


此时，我根据大小对所有的内存分配操作进行排序，并检查所有最小的分配操作。一个突出的部分是在 `curl_multi_wait()` 中，它是一个典型的在 curl 传输主循环中被反复调用的函数。对于大多数典型情况，我将其转换为[使用堆栈](https://github.com/curl/curl/commit/5f1163517e1597339d)。在大量重复的调用函数中避免 malloc 是一件好事。


### 重新计数


现在，如上面的脚本所示，同样的 `curl localhost` 命令从 curl 7.53.1 的 115 次分配操作下降到 80 个分配操作，而没有牺牲任何东西。轻松地有 26％ 的改善。一点也不差！


由于我修改了 `curl_multi_wait()`，我也想看看它实际上是如何改进一些稍微更高级一些的传输。我使用了 [multi-double.c](https://github.com/curl/curl/commit/5f1163517e1597339d) 示例代码，添加了初始化内存记录的调用，让它使用 `curl_multi_wait()`，并且并行下载了这两个 URL：



```
http://www.example.com/
http://localhost/512M

```

第二个文件是 512 兆字节的零，第一个文件是一个 600 字节的公共 html 页面。这是 [count-malloc.c 代码](https://gist.github.com/bagder/dc4a42cb561e791e470362da7ef731d3)。


首先，我使用 7.53.1 来测试上面的例子，并使用 `memanalyze` 脚本检查：



```
Mallocs: 33901
Reallocs: 5
Callocs: 24
Strdups: 31
Wcsdups: 0
Frees: 33956
Allocations: 33961
Maximum allocated: 160385

```

好了，所以它总共使用了 160KB 的内存，分配操作次数超过 33900 次。而它下载超过 512 兆字节的数据，所以它每 15KB 数据有一次 malloc。是好是坏？


回到 git master，现在是 7.54.1-DEV 的版本 - 因为我们不太确定当我们发布下一个版本时会变成哪个版本号。它可能是 7.54.1 或 7.55.0，它还尚未确定。我离题了，我再次运行相同修改的 multi-double.c 示例，再次对内存日志运行 memanalyze，报告来了：



```
Mallocs: 69
Reallocs: 5
Callocs: 24
Strdups: 31
Wcsdups: 0
Frees: 124
Allocations: 129
Maximum allocated: 153247

```

我不敢置信地反复看了两遍。发生什么了吗？为了仔细检查，我最好再运行一次。无论我运行多少次，结果还是一样的。


### 33961 vs 129


在典型的传输中 `curl_multi_wait()` 被调用了很多次，并且在传输过程中至少要正常进行一次内存分配操作，因此删除那个单一的微小分配操作对计数器有非常大的影响。正常的传输也会做一些将数据移入或移出链表和散列操作，但是它们现在也大都是无 malloc 的。简单地说：剩余的分配操作不会在传输循环中执行，所以它们的重要性不大。


以前的 curl 是当前示例分配操作数量的 263 倍。换句话说：新的是旧的分配操作数量的 0.37％ 。


另外还有一点好处，新的内存分配量更少，总共减少了 7KB（4.3％）。


### malloc 重要吗？


在几个 G 内存的时代里，在传输中有几个 malloc 真的对于普通人有显著的区别吗？对 512MB 数据进行的 33832 个额外的 malloc 有什么影响？


为了衡量这些变化的影响，我决定比较 localhost 的 HTTP 传输，看看是否可以看到任何速度差异。localhost 对于这个测试是很好的，因为没有网络速度限制，更快的 curl 下载也越快。服务器端也会相同的快/慢，因为我将使用相同的测试集进行这两个测试。


我相同方式构建了 curl 7.53.1 和 curl 7.54.1-DEV，并运行这个命令：



```
curl http://localhost/80GB -o /dev/null

```

下载的 80GB 的数据会尽可能快地写到空设备中。


我获得的确切数字可能不是很有用，因为它将取决于机器中的 CPU、使用的 HTTP 服务器、构建 curl 时的优化级别等，但是相对数字仍然应该是高度相关的。新代码对决旧代码！


7.54.1-DEV 反复地表现出更快 30％！我的早期版本是 2200MB/秒增加到当前版本的超过 2900 MB/秒。


这里的要点当然不是说它很容易在我的机器上使用单一内核以超过 20GB/秒的速度来进行 HTTP 传输，因为实际上很少有用户可以通过 curl 做到这样快速的传输。关键在于 curl 现在每个字节的传输使用更少的 CPU，这将使更多的 CPU 转移到系统的其余部分来执行任何需要做的事情。或者如果设备是便携式设备，那么可以省电。


关于 malloc 的成本：512MB 测试中，我使用旧代码发生了 33832 次或更多的分配。旧代码以大约 2200MB/秒的速率进行 HTTP 传输。这等于每秒 145827 次 malloc - 现在它们被消除了！600 MB/秒的改进意味着每秒钟 curl 中每个减少的 malloc 操作能额外换来多传输 4300 字节。


### 去掉这些 malloc 难吗？


一点也不难，非常简单。然而，有趣的是，在这个旧项目中，仍然有这样的改进空间。我有这个想法已经好几年了，我很高兴我终于花点时间来实现。感谢我们的测试套件，我可以有相当大的信心做这个“激烈的”内部变化，而不会引入太可怕的回归问题。由于我们的 API 很好地隐藏了内部，所以这种变化可以完全不改变任何旧的或新的应用程序……


（是的，我还没在版本中发布该变更，所以这还有风险，我有点后悔我的“这很容易”的声明……）


### 注意数字


curl 的 git 仓库从 7.53.1 到今天已经有 213 个提交。即使我没有别的想法，可能还会有一次或多次的提交，而不仅仅是内存分配对性能的影响。


### 还有吗？


还有其他类似的情况么？


也许。我们不会做很多性能测量或比较，所以谁知道呢，我们也许会做更多的愚蠢事情，我们可以收手并做得更好。有一个事情是我一直想做，但是从来没有做，就是添加所使用的内存/malloc 和 curl 执行速度的每日“监视” ，以便更好地跟踪我们在这些方面不知不觉的回归问题。


### 补遗，4/23


（关于我在 hacker news、Reddit 和其它地方读到的关于这篇文章的评论）


有些人让我再次运行那个 80GB 的下载，给出时间。我运行了三次新代码和旧代码，其运行“中值”如下：


旧代码：



```
real    0m36.705s
user    0m20.176s
sys     0m16.072s

```

新代码：



```
real    0m29.032s
user    0m12.196s
sys     0m12.820s

```

承载这个 80GB 文件的服务器是标准的 Apache 2.4.25，文件存储在 SSD 上，我的机器的 CPU 是 i7 3770K 3.50GHz 。


有些人也提到 `alloca()` 作为该补丁之一也是个解决方案，但是 `alloca()` 移植性不够，只能作为一个孤立的解决方案，这意味着如果我们要使用它的话，需要写一堆丑陋的 `#ifdef`。




---


via: <https://daniel.haxx.se/blog/2017/04/22/fewer-mallocs-in-curl/>


作者：[DANIEL STENBERG](https://daniel.haxx.se/blog/author/daniel/) 译者：[geekpi](https://github.com/geekpi) 校对：[wxy](https://github.com/wxy)


本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出


OK
Today I landed yet [another small change](https://github.com/curl/curl/commit/cbae73e1dd95946597ea74ccb580c30f78e3fa73) to libcurl internals that further reduces the number of small mallocs we do. This time the generic linked list functions got converted to become malloc-less (the way linked list functions should behave, really).

## Instrument mallocs

I started out my quest a few weeks ago by instrumenting our memory allocations. This is easy since we have our own memory debug and logging system in curl since many years. Using a debug build of curl I run this script in my build dir:

#!/bin/sh export CURL_MEMDEBUG=$HOME/tmp/curlmem.log ./src/curl http://localhost ./tests/memanalyze.pl -v $HOME/tmp/curlmem.log

For curl 7.53.1, this counted about 115 memory allocations. Is that many or a few?

The memory log is very basic. To give you an idea what it looks like, here’s an example snippet:

MEM getinfo.c:70 free((nil)) MEM getinfo.c:73 free((nil)) MEM url.c:294 free((nil)) MEM url.c:297 strdup(0x559e7150d616) (24) = 0x559e73760f98 MEM url.c:294 free((nil)) MEM url.c:297 strdup(0x559e7150d62e) (22) = 0x559e73760fc8 MEM multi.c:302 calloc(1,480) = 0x559e73760ff8 MEM hash.c:75 malloc(224) = 0x559e737611f8 MEM hash.c:75 malloc(29152) = 0x559e737a2bc8 MEM hash.c:75 malloc(3104) = 0x559e737a9dc8

## Check the log

I then studied the log closer and I realized that there were many small memory allocations done from the same code lines. We clearly had some rather silly code patterns where we would allocate a struct and then add that struct to a linked list or a hash and that code would then subsequently add yet another small struct and similar – and then often do that in a loop. (I say *we* here to avoid blaming anyone, but of course I myself am to blame for most of this…)

Those two allocations would always happen in pairs and they would be freed at the same time. I decided to address those. Doing very small (less than say 32 bytes) allocations is also wasteful just due to the very large amount of data in proportion that will be used just to keep track of that tiny little memory area (within the malloc system). Not to mention fragmentation of the heap.

So, fixing the hash code and the linked list code to not use mallocs were immediate and easy ways to remove over 20% of the mallocs for a plain and simple ‘curl http://localhost’ transfer.

At this point I sorted all allocations based on size and checked all the smallest ones. One that stood out was one we made in *curl_multi_wait(),* a function that is called over and over in a typical curl transfer main loop. I converted it over to [use the stack](https://github.com/curl/curl/commit/5f1163517e1597339d) for most typical use cases. Avoiding mallocs in very repeatedly called functions is a good thing.

## Recount

Today, the script from above shows that the same “curl localhost” command is **down to 80 allocations from the 115** curl 7.53.1 used. Without sacrificing anything really. An easy 26% improvement. Not bad at all!

But okay, since I modified curl_multi_wait() I wanted to also see how it actually improves things for a slightly more advanced transfer. I took the [multi-double.c](https://github.com/curl/curl/commit/5f1163517e1597339d) example code, added the call to initiate the memory logging, made it uses curl_multi_wait() and had it download these two URLs in parallel:

http://www.example.com/ http://localhost/512M

The second one being just 512 megabytes of zeroes and the first being a 600 bytes something public html page. Here’s the [count-malloc.c code](https://gist.github.com/bagder/dc4a42cb561e791e470362da7ef731d3).

First, I brought out 7.53.1 and built the example against that and had the memanalyze script check it:

Mallocs: 33901 Reallocs: 5 Callocs: 24 Strdups: 31 Wcsdups: 0 Frees: 33956Allocations: 33961Maximum allocated: 160385

Okay, so it used 160KB of memory totally and it did over 33,900 allocations. But ok, it downloaded over 512 megabytes of data so it makes one malloc per 15KB of data. Good or bad?

Back to git master, the version we call 7.54.1-DEV right now – since we’re not quite sure which version number it’ll become when we release the next release. It can become 7.54.1 or 7.55.0, it has not been determined yet. But I digress, I ran the same modified multi-double.c example again, ran memanalyze on the memory log again and it now reported…

Mallocs: 69 Reallocs: 5 Callocs: 24 Strdups: 31 Wcsdups: 0 Frees: 124Allocations: 129Maximum allocated: 153247

I had to look twice. Did I do something wrong? I better run it again just to double-check. The results are the same no matter how many times I run it…

## 33,961 vs 129

curl_multi_wait() is called a lot of times in a typical transfer, and it had at least one of the memory allocations we normally did during a transfer so removing that single tiny allocation had a pretty dramatic impact on the counter. A normal transfer also moves things in and out of linked lists and hashes a bit, but they too are mostly malloc-less now. Simply put: the remaining allocations are not done in the transfer loop so they’re way less important.

The old curl did **263 times** the number of allocations the current does for this example. Or the other way around: the new one does **0.37%** the number of allocations the old one did…

As an added bonus, the new one also allocates less memory in total as it decreased that amount by 7KB (4.3%).

## Are mallocs important?

In the day and age with many gigabytes of RAM and all, does a few mallocs in a transfer really make a notable difference for mere mortals? What is the impact of 33,832 extra mallocs done for 512MB of data?

To measure what impact these changes have, I decided to compare HTTP transfers from localhost and see if we can see any speed difference. localhost is fine for this test since there’s no network speed limit, but the faster curl is the faster the download will be. The server side will be equally fast/slow since I’ll use the same set for both tests.

I built curl 7.53.1 and curl 7.54.1-DEV identically and ran this command line:

curl http://localhost/80GB -o /dev/null

80 gigabytes downloaded as fast as possible written into the void.

The exact numbers I got for this may not be totally interesting, as it will depend on CPU in the machine, which HTTP server that serves the file and optimization level when I build curl etc. But the relative numbers should still be highly relevant. The old code vs the new.

7.54.1-DEV repeatedly performed **30% faster!** The 2200MB/sec in my build of the earlier release increased to over 2900 MB/sec with the current version.

The point here is of course not that it easily can transfer HTTP over 20 Gigabit/sec using a single core on my machine – since there are very few users who actually do that speedy transfers with curl. The point is rather that curl now uses less CPU per byte transferred, which leaves more CPU over to the rest of the system to perform whatever it needs to do. Or to save battery if the device is a portable one.

On the cost of malloc: The 512MB test I did resulted in 33832 more allocations using the old code. The old code transferred HTTP at a rate of about 2200MB/sec. That equals **145,827 mallocs/second **– that are now removed! A 600 MB/sec improvement means that curl managed to transfer 4300 bytes extra for each malloc it didn’t do, each second.

## Was removing these mallocs hard?

Not at all, it was all straight forward. It is however interesting that there’s still room for changes like this in a project this old. I’ve had this idea for some years and I’m glad I finally took the time to make it happen. Thanks to our test suite I could do this level of “drastic” internal change with a fairly high degree of confidence that I don’t introduce too terrible regressions. Thanks to our APIs being good at hiding internals, this change could be done completely without changing anything for old or new applications.

(Yeah I haven’t shipped the entire change in a release yet so there’s of course a risk that I’ll have to regret my “this was easy” statement…)

## Caveats on the numbers

There have been 213 commits in the curl git repo from 7.53.1 till today. There’s a chance one or more other commits than just the pure alloc changes have made a performance impact, even if I can’t think of any.

## More?

Are there more “low hanging fruits” to pick here in the similar vein?

Perhaps. We don’t do a lot of performance measurements or comparisons so who knows, we might do more silly things that we could stop doing and do even better. One thing I’ve always wanted to do, but never got around to, was to add daily “monitoring” of memory/mallocs used and how fast curl performs in order to better track when we unknowingly regress in these areas.

## Addendum, April 23rd

(Follow-up on some comments on this article that I’ve read on [hacker news](https://news.ycombinator.com/item?id=14177739), [Reddit](https://www.reddit.com/r/programming/comments/671ucd/curl_reducing_malloc3_calls_lead_to_30_faster/) and elsewhere.)

Someone asked and I ran the 80GB download again with ‘time’. Three times each with the old and the new code, and the “middle” run of them showed these timings:

Old code:

real 0m36.705s user 0m20.176s sys 0m16.072s

New code:

real 0m29.032s user 0m12.196s sys 0m12.820s

The server that hosts this 80GB file is a standard Apache 2.4.25, and the 80GB file is stored on an SSD. The CPU in my machine is a core-i7 3770K 3.50GHz.

Someone also mentioned alloca() as a solution for one of the patches, but alloca() is not portable enough to work as the sole solution, meaning we would have to do ugly #ifdef if we would want to use alloca() there.

Have you tried heaptrack? It is pretty neat tool for memory debugging. It doesn’t require custom allocator functions. It has a gui that makes it easy to compare two builds/runs.

No I haven’t, but it looks like a nifty tool I should investigate closer…

When do you do that on Firefox ? 😉

Hehe… I’d say that one of the beauties with working on curl and libcurl is the simplicity. No processes/threads (with ownership, locking, IPC etc), no C++ and not the least: a relatively small code base with a rather focused and narrow use case. It just transfers data…

Wow, 2.9GB/s !? I just come to ~330MB/s here (i3, 3.1GHz, SSD apache Debian stock installation, 5.6GB gzipped test file). The bottleneck here seems to be the SSD. So the test basically measures the speed of my SSD (why the heck is your SSD so much faster ?)

Anyways, a quick comparison with wget2 using ‘time’ shows the same ‘real’ for both tools, but the other differ:

curl 7.54.1-DEV (current master):

user 0m1.272s

sys 0m2.296s

Wget2 (current master):

user 0m0.176s

sys 0m1.844s

So curl has quite some air for further optimizations before it equals out Wget2.

Clearly, and I will never claim that we are ever “done” optimizing or improving.

But a comparison of numbers like that between different projects is not really fair nor easy. I mean, curl uses and libcurl provides an API that sets limitations on how curl can and will perform, for example. In a quite different manner than wget2 does or needs. Not to mention other portability ambitions. Still, curious that the difference was that notable. Makes me wonder what the main explanation is…

Regarding the 80GB file, it is simply a sparse file made with “truncate -s 80G” stored on the SSD. Nothing magic at all.

Thanks for the sparse file trick. Here come the numbers for a 80GB file (best out of several runs). And yes, fair comparisons are difficult. Maybe someone likes to find out about the differences.

Curl (-s and -o /dev/null):

real 0m38.788s

user 0m9.244s

sys 0m19.028s

Wget2 (-O /dev/null):

real 0m38.503s

user 0m0.972s

sys 0m13.052s

One basic thing: I believe wget2 uses a 100KB buffer while curl uses a 16KB one (by default), which explains some of the differences. My numbers for a curl using 100KB buffer:

real 0m26.039s (down from 29)

user 0m4.404s (down from 12)

sys 0m8.156s (down from 12)

That’s an awesome speedup ! You should consider using larger buffers in the curl tool (maybe not in the library).

Yes, agreed!

2200MB/sec and 2900 MB/sec are not 20 Gigabit/s.

2900 megabyte is 23200 megabits. That is more than 20 gigabits. Math!

That sounds great, I wonder if we care. Not only do large file transfers like that rarely happen, as the one user discovered the bottleneck is usually the hard drive.

Even if, lets say, you are doing only processing on the files after you retrieve them, such as passing html to a parser, I would think that parsing process would be pretty slow as well.

My point is, I wonder if the slowness of the malloc makes any real-world difference that we care about?

Sebastian: I already addressed that in the post. I doubt there will be many users noticing a difference, but spending less CPU is still valuable to save power and to allow other parts to use more.

Don’t get me wrong, I don’t think you should revert these changes – they do save battery life.

Put another way though, the slowness of real world operations like disk ops or payload processing also has a corresponding power cost, so again, compared to real world applications, do we really care about the couple microwatt hours we save here compared to the power cost of writing to a drive, or doing complex dom processing?

I would think that would be where the real focus on power conservation would be, the savings this change produces are probably a rounding error in comparison. I’d be interested in seeing the numbers, though.

Hey Daniel,

it’s really nice that you’re doing this. I’m constantly fighting against anyone introducing malloc() calls in haproxy’s fast path for this exact reason. I know that sometimes it’s difficult to avoid them. Often for most objects, using some fixed-size pools significantly helps without changing the code too much. Good luck with the 80 remaining ones, you’re on the right direction!