title: Why is the libcurl multi interface being unreasonably slow on some of my systems?
body: <p>I am trying to write a small application using the libcurl multi interface in C++ and am running into an issue where it performs (much) slower than expected: for 49 requests (arbitrary number) against a static HTML page (served by nginx on a small VM I rent on the internet) it takes a pretty constant 32 seconds.</p>
  <p>Running the application against Google or GitHub takes just as long and benching my static page using <a href="https://github.com/rakyll/hey" rel="nofollow noreferrer">hey</a> yields &gt;1500 requests/s, so I'm pretty sure it's not the servers fault. Looping over the <code>curl</code> command line application 50 times in a row is also quite a bit faster with just 13 seconds total.</p>
  <p>Here is the code in question (minimal code for readability):</p>
  <pre><code>#include &lt;curl/curl.h&gt;
  
  #include &lt;memory&gt;
  #include &lt;string&gt;
  #include &lt;vector&gt;
  
  
  struct Request {
    std::shared_ptr&lt;CURL&gt; request;
  };
  
  
  int request(std::string url, CURLM *curlm, std::vector&lt;Request&gt; &amp;requests) {
    Request r = {
      .request = std::shared_ptr&lt;CURL&gt;(curl_easy_init(), curl_easy_cleanup),
    };
  
    curl_easy_setopt(r.request.get(), CURLOPT_URL, url.c_str());
  
    curl_multi_add_handle(curlm, r.request.get());
    requests.push_back(r);
  
    return 0;
  }
  
  int main(__attribute__((unused)) int argc, char *argv[]) {
    curl_global_init(CURL_GLOBAL_ALL);
  
    std::vector&lt;Request&gt; requests;
    CURLM *multi = curl_multi_init();
  
    // generate an arbitrary amount of requests
    for(int i = 0; i &lt; 49; i++)
      request(std::string(argv[1]), multi, requests);
  
  
    int running = 0;
    curl_multi_perform(multi, &amp;running);
  
    do {
      int numfds = 0;
  
      curl_multi_perform(multi, &amp;running);
  
      // select
      fd_set fdread, fdwrite, fdexcep;
      FD_ZERO(&amp;fdread);
      FD_ZERO(&amp;fdwrite);
      FD_ZERO(&amp;fdexcep);
  
      struct timeval timeout = {
        .tv_sec = 0,
        .tv_usec = 500
      };
  
      curl_multi_fdset(multi, &amp;fdread, &amp;fdwrite, &amp;fdexcep, &amp;numfds);
      select(numfds+1, &amp;fdread, &amp;fdwrite, &amp;fdexcep, &amp;timeout);
  
      // or poll
      //curl_multi_poll(multi, nullptr, 0, 1000, &amp;numfds);
    } while(running &gt; 0);
  
    return 0;
  }
  </code></pre>
  <p>I have tested on multiple client systems: my workstation is an Ubuntu 20.04, the results can be reproduced on CentOS 8, Ubuntu 18.04 and Debian 10 (all libcurl 7.60 series), however on CentOS 7 (libcurl 7.29.0) the same code finishes within 3-5 seconds! Apart from my workstation all of these systems I tested are fresh installed VMs running on the same host with the same network configuration. Meanwhile, the VM on the internet (Debian 10, libcurl 7.64) is about as fast as I expect with sub-second results.</p>
  <p>Using Wireshark I can see all connections to the webserver are opened at once, but the actual HTTP requests are sent in batches with increasing intervals - the last ones are sent as late as ~30 seconds after opening the connection.</p>
  <p>There is no time difference between the <code>select()</code> way of doing things vs using <code>curl_multi_poll()</code> either and I have gone through the cURL documentation looking for options that might increase performance; enabling pipelining does not make any difference nor does disabling HTTP2 and most of the limits (max connections, for example) seem to be set to unlimited by default.</p>
  <p>Now here is the fun part: when I run this application through strace, a run only takes 2-4 seconds per run on any of the local VMs. At first I thought <code>strace</code> might be breaking things, however if I discard the output of <code>strace</code> via shell redirection it looks like the application is working properly, so what gives?</p>
  <p><strong>Why does using <code>strace</code> improve things at all and what may cause run time to be wildly different across these systems?</strong></p>
  <p>Any ideas what I'm missing? My Google-fu has left me this time.</p>