title: nodejs http request stuck
body: <p>I am a beginner in coding.</p>
  
  <p>I want to scrape a lot of webpages, so there is my code.
  The first one takes 2 arguments, host and paths. 
  It will call the second function 'getXml' by forEach, and give it host and path to scrape the webpages one by one. </p>
  
  <p>It usually works, and tell me error message when failing to get the webpage. But sometimes it won't show any error and the program just stuck there. I add two console.log in my second code to see what happens,
      console.log('chunk done');
      console.log('http end');
  When the program stucks again.
  the message on command line is</p>
  
  <pre><code>chunk done
  chunk done
  chunk done
  http end
  scrape webpage done(succeed to scrape a webpage)
  chunk done
  chunk done
  http end
  scrape webpage done (succeed to scape a webpage)
  chunk done
  chunk done
  chunk done
  (somethimes it stucks here, and no message shows, it just don't move on)
  </code></pre>
  
  <p>I don't know what's going on here and how to fix it.
  It's similar this quesion: 
  <a href="https://stackoverflow.com/questions/29103275/request-get-stuck-from-nodejs">request get stuck from nodejs</a> </p>
  
  <p>Is it because the internet connection(untable sometimes)? If it is, how to deal with this situation?</p>
  
  <p>Would somebody give me some help and advise? I will be very appreciate for that. Thank you.</p>
  
  <p>My node version is v7.2.0</p>
  
  <p>And I also use the following modules.</p>
  
  <p>"babel-polyfill": "6.16.0", "babel-preset-latest": "6.16.0", "babel-preset-stage-0": "6.16.0", "babel-register": "6.18.0"</p>
  
  <p>First code:</p>
  
  <pre><code>import {getXml} from './getXml.js';
  import fs from 'fs';
  
  export function scrape(host, paths) {    
    let ready = Promise.resolve(null);
  
    paths.forEach((path, index) =&gt; {
      ready = ready.then(() =&gt; {
        return getXml(host, path);
      }).then((xml) =&gt; {
        // do something with the xml
        // I use fs.writeFileSync to save the xml
        console.log('scrape', host + path, 'done');
      }, (errMessage) =&gt; {
        // do somthing if there is error
        // I use fs.appendFileSync to save the err messages
        console.log('scrape failed:', errMessage);
      });
    });
  };
  </code></pre>
  
  <p>Second code:</p>
  
  <pre><code>import http from 'http';
  
  export function getXml(host, path) {
    return new Promise((resolve, reject) =&gt; {
      let option = {
        "host": host,
        "path": path
      };
  
      http.get(option, (res) =&gt; {
        let xml = '';
        res.setEncoding('utf8');
        res.on('data', (chunk) =&gt; {
          xml += chunk;
          console.log('chunk done');
        });
  
        res.on('end', () =&gt; {
          resolve(xml);
          console.log('http end');
        });
      }).on('error', (err) =&gt; {
        let errMessage = host + path;
        reject(errMessage);
      });
    });
  };
  </code></pre>