title: How to fix puppeteer scrape fail
body: <p>I'd like to save a webpage's HTML code with node.js and puppeteer.
  When I start the program with 'headless-browser: false', I can see that the page loads fully, all the datas are there. But if I try to save the HTML, I get only this: </p>
  
  <pre><code>&lt;!DOCTYPE html&gt;&lt;html&gt;&lt;head&gt;
  &lt;meta name="ROBOTS" content="NOINDEX, NOFOLLOW"&gt;
  &lt;meta http-equiv="cache-control" content="max-age=0"&gt;
  &lt;meta http-equiv="cache-control" content="no-cache"&gt;
  &lt;meta http-equiv="expires" content="0"&gt;
  &lt;meta http-equiv="expires" content="Tue, 01 Jan 1980 1:00:00 GMT"&gt;
  &lt;meta http-equiv="pragma" content="no-cache"&gt;
  &lt;meta http-equiv="refresh" content="10; url=/distil_r_captcha.html?requestId=16a-84c6-42b6-9023-a45b3854e34c&amp;amp;httpReferrer=%2Fli"&gt;
  &lt;script type="text/javascript"&gt;
          (function(window){
                  try {
                          if (typeof sessionStorage !== 'undefined'){
                                  sessionStorage.setItem('distil_referrer', document.referrer);
                          }
                  } catch (e){}
          })(window);
  &lt;/script&gt;
  &lt;script type="text/javascript" src="/elrhculcipoedjwh.js" defer=""&gt;&lt;/script&gt;&lt;style type="text/css"&gt;#d__fFH{position:absolute;top:-5000px;left:-5000px}#d__fF{font-family:serif;font-size:200px;visibility:hidden}#xaqctssquudxqdqxzveurrreayw{display:none!important}&lt;/style&gt;&lt;/head&gt;
  &lt;body&gt;
  &lt;div id="distilIdentificationBlock"&gt;&amp;nbsp;&lt;/div&gt;
  &lt;/body&gt;&lt;/html&gt;
  </code></pre>
  
  <p>So I'm a little bit confused: If the webpage knows that the request came from a robot (=so I can only download this blocked HTML code) then why the content shows up? 
  Or from the other perspective: 
  If the webpage doesn't know that the request came from a robot (=so the content shows up) then why can I only download this blocked HTML?</p>
  
  <p>My code: </p>
  
  <pre><code>const puppeteer = require('puppeteer');
  
  (async () =&gt; {
  
      const browser = await puppeteer.launch({ headless: false });
      const context = await browser.createIncognitoBrowserContext();
      const page = await context.newPage();
  
      await page.evaluateOnNewDocument(() =&gt; {
          Object.defineProperty(navigator, 'webdriver', {
              get: () =&gt; false,
          });
      });
  
      await page.evaluateOnNewDocument(() =&gt; {
          window.navigator.chrome = {
              runtime: {},
          };
      });
  
      await page.evaluateOnNewDocument(() =&gt; {
          const originalQuery = window.navigator.permissions.query;
          return window.navigator.permissions.query = (parameters) =&gt; (
              parameters.name === 'notifications' ?
                  Promise.resolve({
                      state: Notification.permission
                  }) :
                  originalQuery(parameters)
          );
      });
  
      await page.evaluateOnNewDocument(() =&gt; {
          Object.defineProperty(navigator, 'plugins', {
              get: () =&gt; [1, 2, 3, 4, 5],
          });
      });
  
      await page.evaluateOnNewDocument(() =&gt; {
          Object.defineProperty(navigator, 'languages', {
              get: () =&gt; ['en-EN', 'en'],
          });
      });
  
      await page.setViewport({
          'width': 1024,
          'height': 768,
          'deviceScaleFactor': 1,
          'isMobile': false,
          'hasTouch': false,
          'isLandscape': false
      });
  
      await page.setUserAgent('Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36');
      await page.goto(url, { waitUntil: 'load' });
      const html = await page.content();
      console.log(html);
      await browser.close();
  })();
  </code></pre>
  
  <p>How can I sole this problem? Maybe I try to save the HTML code too early?
  Thanks in advance.</p>