title: Node.js performance optimization involving HTTP calls
body: <p>I have a Node.js application which opens a file, scans each line and makes a REST call that involves Couchbase for each line.  The average number of lines in a file is about  12 to 13 million. Currently without any special settings my app can completely process ~1 million records in ~24 minutes.  I went through a lot of questions, articles, and Node docs but couldn't find out any information about following:</p>
  
  <ol>
  <li>Where's the setting that says node can open X number of http connections / sockets concurrently ? and  can I change it? </li>
  <li>I had to regulate the file processing because the file reading is much faster than the REST call so after a while there are too many open REST requests and it clogs the system and it goes out of memory... so now I read 1000 lines wait for the REST calls to finish for those and then resume it ( i am doing it using pause and resume methods on stream)  Is there a better alternative to this? </li>
  <li>What all possible optimizations can I perform so that it becomes faster than this. I know the gc related config that prevents from frequent halts in the app. </li>
  <li>Is using "cluster" module recommended? Does it work seamlessly? </li>
  </ol>
  
  <p>Background:  We have an existing java application that does exactly same by spawning 100 threads and it is able to achieve slightly better throughput than the current node counterpart. But I want to try node since the two operations in question (reading a file and making a REST call for each line) seem like perfect situation for node app since they both can be async in node where as Java app makes blocking calls for these...</p>
  
  <p>Any help would be greatly appreciated...</p>