flâneur

Crawlee · The scalable web crawling, scraping and automation library for JavaScript/Node.js | Crawlee

crawlee.dev · 1,353 words · saved by 1 readers

JavaScript is the language of the web. Crawlee builds on popular tools like Playwright, Puppeteer and cheerio, to deliver large-scale high-performance web scraping and crawling of any website. Works best with TypeScript! Run headless Chrome, Firefox, WebKit or other browsers, manage lists and queues of URLs to crawl, run crawlers in parallel at maximum system capacity. Handle storage and export of results and rotate proxies. Crawlee can be used stand-alone on your own systems or it can run as a serverless microservice on the Apify Platform. All the crawlers are automatically scaled based on available system resources using the AutoscaledPool class. Advanced options are available to fine-tune scaling behaviour. Never get blocked with unique fingerprints for browsers generated based on real world data. Crawl using HTTP requests as if they were from browsers, using auto-generated headers based on real browsers and their TLS fingerprints. There are three main classes that you can use to st

# Crawlee for JavaScript · Build reliable crawlers. Fast. - [Build reliable web scrapers. Fast.](https://crawlee.dev/index.md) ## blog - [Crawlee Blog - learn how to build better scrapers](https://crawlee.dev/blog.md) - [Archive](https://crawlee.dev/blog/archive.md) - [Authors](https://crawlee.dev/blog/authors.md) - [Building a Gradcracker scraper with Crawlee: anti-bot failures and redirect bugs](https://crawlee.dev/blog/building-gradcracker-scraper-with-crawlee.md) - [Current problems and mistakes of web scraping in Python and tricks to solve them!](https://crawlee.dev/blog/common-problems-i

saved by

related reading