How is Firecrawl different from traditional scrapers?

Firecrawl is built for AI, covers 96% of the web (including JS-rendered pages), handles proxies and anti-bot mechanisms automatically, and outputs LLM-ready Markdown or JSON in under 1 second. Traditional scrapers require manual proxy management and data cleaning.

How many credits does the free plan include?

The free plan includes 500 credits (scrape 500 pages). After that, upgrade to Hobby, Standard, or Growth plans for more credits and higher rate limits.

Does Firecrawl support JavaScript-rendered websites?

Yes. Firecrawl covers 96% of the web, including JavaScript-heavy pages, without needing Puppeteer or proxy management. It intelligently waits for dynamic content to load.

What's the difference between open-source and hosted versions?

The hosted version includes the proprietary Fire-engine scraper (handles proxies and anti-bot), Actions for page interactions, analytics dashboard, and is one API call away. The open-source version is self-hostable but lacks these features.

Does Firecrawl respect robots.txt?

Yes. Firecrawl's crawl endpoint respects robots.txt rules for the 'FirecrawlAgent' directive. If you notice unexpected behavior, contact help@firecrawl.com.

Firecrawl

Firecrawl is a web data API for AI, offering scraping, crawling, and search. Converts websites to Markdown or structured data, handles JS-rendered pages, covers 96% of the web, delivers in under 1 second.

Visit Firecrawl

Firecrawl

Summary

Firecrawl is a web data API designed for AI applications, converting websites into LLM-ready Markdown or JSON. Solves the problem of traditional scrapers failing on JavaScript-rendered content, complex proxy management, and time-consuming data cleaning.

What is Firecrawl?

Firecrawl is a web scraping and crawling platform built for LLM engineers, data scientists, and AI developers. It offers Scrape (single-page extraction), Crawl (full-site crawling), Search (search and scrape), and Browser Sandbox (browser environment for agents). Covers 96% of the web, including JavaScript-heavy pages, without managing proxies or anti-bot mechanisms. Supports PDF and DOCX parsing, delivers results in under 1 second, ideal for real-time AI apps.

Core Capabilities

Scrape: Convert single pages to Markdown, JSON, or screenshots; supports custom headers and actions (click, scroll, type)
Crawl: Scrape entire websites and all accessible subpages, no sitemap required
Search: Search the web and extract full content, integrating search and scraping
Browser Sandbox: Secure, managed browser environment for AI agents
Document parsing: Extract content from web-hosted PDFs, DOCX, and more
Smart wait: Automatically waits for dynamic content to load, improving reliability
Caching control: Selective caching with customizable patterns
Actions: Perform clicks, scrolls, typing, and other interactions before extraction

Pros

Covers 96% of the web, including JS-rendered pages, without managing proxies or Puppeteer
Delivers results in under 1 second, suitable for real-time AI agents and dynamic apps
Outputs LLM-ready Markdown or structured JSON, reducing token usage
Open-source and transparent (87,900+ GitHub stars), self-hostable or hosted
Integrates with Claude Code, Cursor, Windsurf, and other AI coding tools via one command

Cons

Free plan limited to 500 credits (500 pages), paid subscription required after
Does not support social media platforms (e.g., Facebook, Twitter)
Advanced features (FIRE-1 agent) billed even if requests fail
Credits do not roll over to next month (except auto-recharge and annual enterprise plans)
Self-hosted version lacks Fire-engine proprietary scraper and anti-bot mechanisms

Decision Guidance

Use Firecrawl when: You need to extract large volumes of web data for LLM apps, AI agents, research, or lead enrichment, especially from JavaScript-rendered sites or requiring real-time data. Ideal for LLM engineers, data scientists, and AI researchers.

Consider alternatives when: You only need to scrape static sites, have a very limited budget (paid after 500 free pages), need to scrape social media platforms, or require credit rollover. Traditional scrapers or custom solutions may suffice.