title: how to overcome the limit of scraping data from website
body: <p>I am new to the concept of scraping data off the internet and need some help.</p>
  
  <p>I am using python 3.6.1 to scrape product details from Paytm(e-commerce website in India).</p>
  
  <p>I am using the following webpage url to scrape the data
  <a href="https://paytm.com/shop/g/electronics/computers-accessories/computer-components/laptop-adapters?src=1&amp;q=graphic%20card" rel="nofollow noreferrer">https://paytm.com/shop/g/electronics/computers-accessories/computer-components/laptop-adapters?src=1&amp;q=graphic%20card</a></p>
  
  <p>Problem: The website contains 49 products in a page but i am able to scrape only 30 products. I have also tried for webpage containing mobile phones on the paytm but still i am able to scrape only 30 whereas number of phones in page is 128.</p>
  
  <p>My python code:</p>
  
  <pre><code>from bs4 import BeautifulSoup as soup
  from urllib.request import urlopen as ureq
  my_url='https://paytm.com/shop/g/electronics/mobile-accessories/mobiles/smart-phones?src=1&amp;q=mobile%20phones'
  page=ureq(my_url).read()
  page_soup=soup(page,"html.parser")
  containers=page_soup.find_all("div",{"class":"_2i1r"})
  print(len(containers))
  f=open("paytm_mobiles.csv","w")
  f.write("Product_Name, Amount\n")
  
  for i in containers:
      name=i.find_all("div",{"class":"_2apC"})
      print("Name :" + name[0].text)
      price=i.findAll("span", {'class':"_1kMS"})
      print("Price :"+ price[0].text)
      f.write(name[0].text.replace("."," ")+","+price[0].text+"\n")
  
  f.close()
  </code></pre>
  
  <p>Please help me to overcome the problem.</p>