Firecrawl for Convex lets you scrape, map, and search the web from Convex actions and run durable site crawls tracked in your Convex database with reactive quer
npm install @firecrawl/firecrawl-convexFirecrawl for Convex exposes search, scrape, map, and durable crawl operations as Convex-native functions, backed by the Firecrawl v2 REST API. One-shot operations run directly from Convex actions, while crawls store progress and pages in your Convex database so clients subscribe reactively rather than polling. Webhook delivery from Firecrawl advances crawl state in real time, with a poll-based watchdog as fallback and a completion callback fired as an internal mutation when the crawl finishes.
The firecrawl-convex component exposes a FirecrawlClient with a scrape method you call directly from a Convex action. It accepts a URL and options like formats, onlyMainContent, and maxAge, and returns markdown, HTML, screenshots, or structured JSON extracted via a prompt. No Node.js runtime is needed since everything runs in the standard Convex runtime against the Firecrawl v2 REST API.
The firecrawl-convex component provides a startCrawl method that registers a crawl with Firecrawl, stores a crawls row and a pages table in your Convex database, and advances them via webhooks or polling. Your client subscribes to live crawl progress using useQuery and usePaginatedQuery against plain Convex queries, so no polling is needed in your frontend. When the crawl finishes, an internal mutation you provide is called exactly once with the final status and any context you passed at start time.
The firecrawl-convex search method runs a web search from any Convex action and optionally scrapes each result page in the same call using scrapeOptions. Results are returned as-is from the Firecrawl v2 API with TypeScript types, so new response fields are available as soon as Firecrawl ships them without waiting for a package update.
Pass an onComplete internal mutation to startCrawl and include a context payload such as a userId or document ID. When the crawl reaches a terminal state, the mutation runs exactly once and receives the crawlId, final status, page count, and your context. From there you can schedule further work such as calling an embedding pipeline using ctx.scheduler.runAfter, and you can read the stored pages using firecrawl.listPages inside that pipeline.
Yes. Because a local Convex deployment is not reachable from Firecrawl's servers, you pass mode: 'poll' to startCrawl so the component polls the Firecrawl status endpoint instead of waiting for webhooks. The component also ships a mock Firecrawl server in the example directory that delivers signed webhook events to a local deployment so you can test full crawl flows without spending API credits.
The firecrawl-convex component budgets each page document in UTF-8 bytes against Convex's 1MB document limit. Text and link lists are truncated to fit, while a screenshot, extracted JSON blob, or changeTracking data that does not fit is dropped entirely. Any truncation sets a truncated flag on the page document. Pages that still cannot be stored are counted in an unstored field on the crawl row and in the onComplete callback payload, so the situation is never silent. For very large crawls, you can pass storeContent: false to record only URLs and metadata.
The component validates each incoming webhook delivery using two checks: an HMAC signature verified against the FIRECRAWL_WEBHOOK_SECRET environment variable when it is set, and a per-crawl token the component generates and passes to Firecrawl when registering the webhook. A delivery that fails either check is rejected with a 401 response and nothing is written to the database.
The firecrawl-convex component throws ConvexErrors carrying a structured payload with code, status, path, and message fields. You can branch on error.data.status to handle specific cases such as 402 for out of credits or 429 for rate limiting. Transient failures including 408, 425, 429, and 5xx responses are retried automatically up to three times with backoff, and the component respects the Retry-After header when present.
Yes. Set the FIRECRAWL_API_URL environment variable in the component configuration and point it at your self-hosted Firecrawl instance. The component will direct all API requests there instead of the default Firecrawl cloud endpoint. Everything else including webhook validation, polling, and the TypeScript client API works the same way.