title: Curl slow multithreading dns
body: <p>The program is made in C++, and it indexes webpages, so all domains are random domain names from the web. The strange part is that the dns <code>fail</code>/<code>not found</code> percentage is small (>5%).</p>
  
  <p>here is the pmp stack trace:</p>
  
  <pre><code>   3886 __GI___poll,send_dg,buf=0xADDRESS,__libc_res_nquery,__libc_res_nquerydomain,__libc_res_nsearch,_nss_dns_gethostbyname3_r,gaih_inet,__GI_getaddrinfo,Curl_getaddrinfo_ex
      601 __GI___poll,Curl_socket_check,waitconnect,singleipconnect,Curl_connecthost,ConnectPlease,protocol_done=protocol_done@entry=0xADDRESS),Curl_connect,connect_host,at
      534 __GI___poll,Curl_socket_check,Transfer,at,getweb,athread,start_thread,clone,??
      498 nanosleep,__sleep,athread,start_thread,clone,??
       50 __GI___poll,Curl_socket_check,Transfer,at,getweb,getweb,athread,start_thread,clone,??
       15 __GI___poll,Curl_socket_check,Transfer,at,getweb,getweb,getweb,athread,start_thread,clone
        7 nanosleep,usleep,main
  </code></pre>
  
  <p>Why are there so many threads at <code>_nss_dns_gethostbyname3_r</code>? What could I do to speed it up.</p>
  
  <p>Could it be because I'm using curl's default synchronous DNS resolver with <code>CURLOPT_NOSIGNAL</code>?</p>
  
  <p>The program is running on a intel I7 (8 cores HT), 16GB ram, Ububtu 12.10.</p>
  
  <p>The bandwidth varies from of 6MB/s (ISP limit) -> 2MB/s at an irregular interval, and it sometimes even drops to a few 100KB/s. </p>