title: Puppeteer serving different pages for the same URL Headless vs Headful
body: <p>I'm not exactly sure why this happens but, I'm trying to scrape the results from Google search results and the results are dynamically loaded as you scroll down. I'm trying to simulate that by scrolling down however when puppeteer is running in headless mode, it doesn't load all the search results whereas when headless is turned off, it returns the correct number of results.</p>
  
  <p>I found that the information I'm looking for is stored in divs with class 'bkWMgd' as you can see there are 7 divs in total:</p>
  
  <p><a href="https://i.stack.imgur.com/v18n9.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/v18n9.png" alt="enter image description here"></a></p>
  
  <p>Puppeteer code:</p>
  
  <pre class="lang-js prettyprint-override"><code>(async (searchQuery) =&gt; {
      const browser = await puppeteer.launch({
          //headless: false
      });
  
      const page = await browser.newPage();
  
      await page.goto('https://google.com/search?q='+searchQuery, { waitUntil: 'networkidle2' });
  
      //Wait for one of the div classes to load
      await page.waitForSelector('div[class=bkWMgd]');
  
  
      //Scroll all the way down
      await page.evaluate(() =&gt; window.scrollTo(0,document.body.scrollHeight));
  
      //Counts how many div with class 'bkWMgd' there are
      const resp = await page.evaluate(() =&gt; document.querySelectorAll('div[class=bkWMgd]').length);
  
      console.log(resp);
  
      await page.screenshot({path: 'example.png'});
  
      await browser.close();
  
  })('cats')
  </code></pre>
  
  <p>Running this code in headless mode console.logs <code>3</code> and running it with headless mode turned off outputs the correct number <code>7</code></p>
  
  <p>These are the screenshots that are captured:</p>
  
  <p>Headless Mode:
  <a href="https://i.stack.imgur.com/PGwZ5.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/PGwZ5.png" alt="enter image description here"></a></p>
  
  <p>Headless Mode OFF: 
  <a href="https://i.stack.imgur.com/8KPMM.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8KPMM.png" alt="enter image description here"></a></p>
  
  <p>For some reason, It seems that different results are being outputted.  I tried creating an IngocnitoContext:</p>
  
  <pre class="lang-js prettyprint-override"><code>    const context = await browser.createIncognitoBrowserContext();
  
      const page = await context.newPage();
  </code></pre>
  
  <p>However, it still leads to different results, therefore I was just wondering if there was a way to ensure that both headless and headful produce the same results. The results I'm interested in are the ones that being produced by when headless is turned off </p>