Loading components
TinyFish
TinyFish lets Convex functions search the web and fetch pages as Markdown, with durable background jobs that store progress and results in your database.
Installation
npm install @tiny-fish/convexAbout TinyFish
TinyFish is a Convex component that wraps the TinyFish Search and Fetch APIs, letting you query the web and retrieve pages as Markdown, HTML, or JSON from your Convex functions. It provides both one-shot actions for immediate results and durable background jobs that store progress and page content in your Convex database, enabling reactive UI subscriptions via standard Convex queries rather than client-side polling.
Benefits
- Run reactive web research jobs from mutations so starting a job and recording ownership commit atomically in the same transaction
- Subscribe to live job progress and fetched pages through standard Convex queries without polling or external state management
- Fetch up to 500 URLs per job with automatic retries, backoff, and a watchdog that recovers jobs across deploys
- Receive partial results cleanly so one failed or bot-blocked URL never drops the rest of a batch
Use cases
how to scrape web pages and store results in Convex database
TinyFish provides a startFetch method callable from a Convex mutation that kicks off a background job, stores each fetched page as Markdown in your database, and fires an onComplete callback when done. Pages are accessible immediately as they land via tinyfish.listPages, so your UI can reactively display results without polling.
web search in Convex actions for RAG or AI agents
TinyFish exposes a search method for use in Convex actions that returns ranked web, news, or research-paper results with URLs and snippets. You can combine it with fetch to retrieve full page content, or use startSearchAndFetch to run both steps as a single durable job that preserves search rank and snippets alongside the fetched Markdown.
durable background fetch jobs in Convex with status tracking
TinyFish durable jobs store a jobs row and a pages table inside your Convex deployment. The getJob query returns structured status fields including queued, running, completed, failed, and cancelled states plus counts of completed and failed URLs. Because these are plain Convex queries, any client subscribed to them receives live updates as the job progresses.
how to research a topic end to end from a Convex mutation
The startSearchAndFetch method accepts a query string, a maxResults count, and optional fetch and search options, then runs a search and reads the top results as one durable job. Each stored page retains its search rank and snippet. The method can be called from a mutation, so starting the job and inserting an ownership record into your own tables happen atomically.
Frequently asked questions
What is the difference between fetch and startFetch in the TinyFish Convex component?
fetch is a synchronous action call that accepts up to 10 URLs and returns their Markdown content directly to the caller. startFetch is a mutation-compatible method that accepts up to 500 URLs, runs the fetching in the background as a durable job, and stores each page in your Convex database as it lands. Use fetch when the caller can wait and the URL count is small; use startFetch for larger batches or work that must outlive the originating request.
How does TinyFish handle authentication and API key security in Convex?
TinyFish reads the API key from the TINYFISH_API_KEY environment variable set on your Convex deployment using npx convex env set. The key is passed to TinyFish only as an X-API-Key header on outbound requests and is never stored in a Convex table or returned to clients. You create the key at agent.tinyfish.ai/api-keys and wire it through the typed component env defined in convex.config.ts.
How do I authorize users before they can trigger a TinyFish search or fetch in Convex?
Convex components cannot access ctx.auth, so authorization belongs in your own wrapper functions. You write a Convex action or mutation that calls requireUser or an equivalent auth check before invoking tinyfish.search or tinyfish.startSearchAndFetch. The TinyFish README and example app demonstrate this pattern explicitly, noting that exported Convex functions are public by default and must gate access before spending your API allowance.
What happens when a URL fails to fetch in a TinyFish durable job?
Failed URLs are stored as pages with status: failed and the TinyFish error code in the error field, so successfully fetched pages are never lost. The job itself is only marked failed when no URLs could be fetched at all, or when TinyFish rejected the request outright due to an invalid key or exhausted allowance. Partial failures are surfaced through listPages by passing status: failed to filter them separately from successful pages.
How do I clean up TinyFish jobs and pages in Convex to control storage usage?
TinyFish jobs and their pages are retained indefinitely until you call tinyfish.deleteJob. Because a page document can be close to 1MB, the README recommends deleting jobs once you have copied the content you need, either at the end of the onComplete callback or from a Convex cron job that removes jobs older than a chosen threshold. You can also limit page size upfront by setting maxCharacters or by passing storeContent: false to store only URL metadata without page text.
Links
Components
Open-source building blocks for your Convex app
Browse components in this category.