Convex Evalbench adds LLM tracing, versioned eval datasets, and regression scoring to your Convex deployment with no extra infrastructure.
npm install convex-evalbenchconvex-evalbench is a Convex component that adds LLM observability, eval execution, and regression testing directly inside your Convex deployment. It records spans from agents and LLM calls into a reactive trace tree, runs versioned datasets against a target action with configurable scorers (exact match, JSON Schema, LLM-as-judge, embedding similarity, custom), and exposes live counters and per-item results via Convex subscriptions with no polling required.
Convex Evalbench provides a withEvalbench adapter that wraps a convex-dev/agent Agent instance and records each LLM call as a span linked to a root agent_step span. Spans share a traceId per operation and stream into a reactive span tree via Convex subscriptions. Raw prompt and completion content is opt-in via the recordContent flag.
Convex Evalbench lets you create a versioned dataset with evalbench.createDataset, then call evalbench.startRun with a target action and scorer config. The runner dispatches each dataset item to your action with bounded concurrency and writes scored results back. You subscribe to evalbench.runSummary for live pass rate counters and evalbench.listResults for per-item scores and trace links.
Convex Evalbench includes an llmAsJudge scorer that accepts a rubric and calls your own LLM action to produce a verdict, which is recorded as a judge span. A multi-judge consensus scorer is also available for quorum-based decisions. Each judging call is traced inside the same Convex deployment.
Convex Evalbench ships an otlpTraceHandler that you mount as a Convex httpAction at a path like /v1/traces. Any OTLP/JSON exporter can then stream spans to it, which are mapped via GenAI semantic conventions and stored in your Convex deployment alongside agent-native spans.
No. Convex Evalbench runs entirely inside your own Convex deployment. Spans, datasets, eval results, and dashboard state are all stored in Convex tables. Reactivity is provided by Convex subscriptions, not polling, and there is no external eval platform or tracing backend required.
Convex Evalbench only stores raw input and output when you explicitly pass them to recordSpan or enable recordContent on the agent adapter. Content at or below 4 KB is stored inline. Larger content is offloaded to Convex File Storage and resolved on demand via the spanContent query, so reactive span tree updates stay small.
Convex Evalbench supports three ingestion paths: a source-agnostic recordSpan API you call directly, the withEvalbench adapter for convex-dev/agent, and evalbenchMiddleware for Vercel AI SDK via wrapLanguageModel. An OTLP/JSON HTTP receiver is also included for any OpenTelemetry-instrumented application.
Convex Evalbench ships exactMatch and jsonSchema as deterministic built-in scorers that run in Convex's V8 runtime. For semantic scoring it provides llmAsJudge with rubric-based judging, a consensus scorer for multi-judge quorum, and embeddingSimilarity against a host embedder action. Custom scorers are supported via defineScorer.
Convex Evalbench includes evalbench.redriveRun which replays incomplete items in a run up to a configured attempt cap. Retryable target failures are retried with exponential backoff. This allows recovery from transient LLM errors or action timeouts without restarting the entire run.