We would like to crawl a vehicle ads website. This website has 400,000 used vehicle ads, and about 20,000 new ads per day. We want to setup something to crawl the 400K ands, and update it every day with the 20K new ads.
(all pages with used cars ads).
The tricky part is crawling the phone number. It is behind the “Voir le numéro” button, which sends a POST request. It looks like the POST request needs the same IP as the one used to access the ad page.
When trying to do it, we are blocked by the website.
We tried to do it without proxy : we are blocked after 10 calls if we go too fast.
Previous: What’s the best way of scraping data from a website?