Bright Data Datasets

Trigger Bright Data dataset collections from Convex actions and reactively subscribe to structured records via useQuery with zero webhook infrastructure.

Installation

npm install @sholajegede/convex-bright-data-datasets

About Bright Data Datasets

Drop-in Convex component that wraps Bright Data's Datasets API end to end. Trigger async collections for LinkedIn, Amazon, job postings, and 120+ other datasets from a Convex action. The component mounts a webhook handler, receives NDJSON delivery, parses and stores each record individually in component-owned tables, and updates snapshot status reactively. Frontend subscribes via useQuery with no polling, no custom webhook infrastructure, and no storage layer to build. Includes synchronous scrap

Benefits

Eliminate custom webhook infrastructure with a drop-in HTTP handler that receives and parses Bright Data results automatically
Subscribe to dataset records reactively via useQuery with real-time updates as webhook batches arrive
Trigger LinkedIn, Amazon, job posting, and 120+ other dataset collections directly from Convex actions
Track collection status and progress automatically with pending-to-ready state management

Use cases

how to scrape LinkedIn profiles in Convex

Use the Bright Data Datasets component to trigger LinkedIn profile collections from Convex actions. Call brightDatasets.trigger() with the LinkedIn dataset ID and profile URLs, then subscribe to results via useQuery as the webhook delivers structured profile data.

webhook handler for dataset API in Convex

Mount the component's webhook handler in convex/http.ts using createWebhookHandler(). It automatically receives Bright Data NDJSON deliveries, parses records, stores them in component tables, and updates snapshot status reactively.

real-time data scraping with Convex

Trigger async dataset collections with brightDatasets.trigger() and subscribe to records via useQuery. Your React components re-render automatically as webhook batches arrive, providing real-time updates without polling.

Frequently asked questions

What datasets can I collect with this component?

The Bright Data Datasets component supports 120+ datasets including LinkedIn profiles, Amazon products, Instagram posts, job listings, Airbnb, Zillow, Google Maps, and more. You need a Bright Data dataset ID (format gd_...) from their Web Scraper API.

How do I get results from dataset collections in real-time?

The component stores snapshot metadata immediately when you trigger a collection, then updates records reactively as the webhook receives results. Subscribe via useQuery to get automatic re-renders when data arrives, with no polling required.

Can I scrape data synchronously without webhooks?

Yes, use brightDatasets.scrape() for small single-URL jobs that complete within 1 minute. For longer jobs, it automatically falls back to the async webhook flow with snapshot polling.

How do I handle webhook delivery in my Convex app?

Mount the webhook handler in convex/http.ts using createWebhookHandler(components.convexBrightDataDatasets). The handler automatically parses NDJSON deliveries, stores records in component tables, and updates snapshot status.