Crawl entire websites with a single API call using Browser Rendering · Changelog
Browser Rendering's new /crawl endpoint lets you submit a starting URL and automatically discover, render, and return content from an entire website as HTML, Markdown, or structured JSON.
Edit: this post has been edited to clarify crawling behavior with respect to site guidance. You can now crawl an entire website with a single API call using Browser Rendering's new /crawl endpoint, available in open beta. Submit a starting URL, and pages are automatically discovered, rendered in a headless browser, and returned in multiple formats, including HTML, Markdown, and structured JSON. The endpoint is a verified bot (intermediary agent) that respects robots.txt and AI Crawl Control ↗ by default, making it easy for developers to comply with website rules, and making it less likely…
saved by
related reading
- Crawlee · The scalable web crawling, scraping and automation library for JavaScript/Node.js | Crawleecrawlee.dev
- How to Build a Web Crawler in Pythonzenrows.com
- Exa | Web Search API, AI Search Engine, & Website Crawlermetaphor.systems
- Introducing Kitesurf: The agent-first browser that runs in V8 isolates on Cloudflare Workersblog.cloudflare.com
- Introducing pay per crawl: Enabling content owners to charge AI crawlers for access | The Cloudflare Blogblog.cloudflare.com
- Building a web search engine from scratch in two months with 3 billion neural embeddingsblog.wilsonl.in
- Web Scraping and Crawling Are Perfectly Legal, Right?benbernardblog.com
- Crawling a billion web pages in just over 24 hoursandrewkchan.dev
- Monolith – CLI tool for saving complete web pages as a single HTML filenews.ycombinator.com
- Browserbasesubstack.com
- GitHub - assafelovic/gpt-researcher: An autonomous agent that conducts deep research on any data using any LLM providersgithub.com
- Sandbox SDKsandbox.cloudflare.com