Show HN: Crawlee for Python – a web scraping and browser automation library
Hey all,<p>This is Jan, the founder of Apify (<a href="https:&#x2F;&#x2F;apify.com&#x2F;" rel="nofollow">https:&#x2F;&#x2F;apify.com&#x2F;</a>) — a full-stack web scraping platform. After the success of Crawlee for JavaScript (<a href="https:&#x2F;&#x2F;github.com&#x2F;apify&#x2F;crawlee&#x2F;">https:&#x2F;&#x2F;github.com&#x2F;apify&#x2F;crawlee&#x2F;</a>) and the demand from the Python community, we&#x27;re launching Crawlee for Python today!<p>The main features are:<p>- A unified programming interface for both HTTP (HTTPX with BeautifulSoup) &amp; headless browser crawling (Playwright)<p>- Automatic parallel crawling based on available system resources<p>- Written in Python with type hints for enhanced developer experience<p>- Automatic retries on errors or when you’re getting blocked<p>- Integrated proxy rotation and session management<p>- Configurable request routing - direct URLs to the appropriate handlers<p>- Persistent queue for URLs to crawl<p>- Pluggable storage for both tabular data and files<p>For details, you can read the announcement blog post: <a href="https:&#x2F;&#x2F;crawlee.dev&#x2F;blog&#x2F;launching-crawlee-python" rel="nofollow">https:&#x2F;&#x2F;crawlee.dev&#x2F;blog&#x2F;launching-crawlee-python</a><p>Our team and I will be happy to answer here any questions you might have.