Agent evals

The backend platform your agents can’t break

Agents should build what you asked for, correctly. You can catch frontend bugs in a browser, but the real failures, the ones that surface at load on launch day, are much harder to find. In our research, frontier models often write build systems that fail at scale, all the while claiming they pass code review with flying colors.

To test this, we asked several agents (Opus 4.8 and Fable from Anthropic, as well as GPT-5.6-Sol and GPT-5.6-Terra from OpenAI) to one-shot the same ten apps on Postgres, then again on Convex.

9 out of 10 apps silently corrupted data on Postgres. None of them did on Convex.

Zero writes lost on Convex

Database updates executed correctly, over time

More users, more problems

Legacy relational databases are full of subtle traps at scale. One of the apps the agent wrote managed shopping carts, but on Postgres the more shoppers buy at once, the more updates are silently dropped. The same app built on Convex stays correct as users scale.

The apps, the load tests, and the measured data →

A verbatim example prompt

"Hey, I run a small online shop and need a backend, please.

Catalog. Each item has a name, a price, and how many are in stock.

Buy. A customer buys an item. After a purchase, the item's remaining stock goes down by one, and we keep a record of the sale.

... Build the whole backend: schema, seed a few items (some with only 1 or 2 in stock), and an API. However you think is best."

Nothing about locks, transactions, or races. Just “let people buy things”.

The real test

Correct by default

Under load, most of the Postgres apps fell over or silently corrupted data, all while answering a cheerful HTTP 200. The Convex apps had none of these issues.

Why the split? On Convex the exact same code is correct by default: every mutation runs as one serializable transaction and retries on conflict, so the first thing an agent reaches for just works, no matter how many requests hit at once. On Postgres it's the opposite default: quietly wrong under load, unless someone knows to add the right lock or constraint to every single operation by hand.

Comparison of Postgres and Convex app correctness under concurrent load. Expand a failing Postgres result for details.
App under concurrent load	Postgres, as generated	Convex, correct by default
Bill splitting 2 Edits at once	Edit lost data lost	Correct by default

Booking Overlapping times • 2 requests	2 Booked double-booked	Correct by default
double-booked Postgres Two appointments whose times overlap both clear the availability check before either commits — the clash guard locks nothing when it finds no conflict — so one stylist is booked for two customers at once. Double-booked. Correct Convex Each booking runs as one serializable transaction that reads the overlapping time range; a competing appointment that overlaps conflicts on that read and retries, so a stylist can't be booked twice. Correct by default.
Kanban 10 Moves • One slot	8 Collide order corrupted	Correct by default

Ledger 10 Entry • 2 Reversals	Reversed 8x corrupted records	Correct by default

Marketplace Add item during checkout	Item lost unpaid & gone	Correct by default

Q&A forum 10 Votes at once	Score 3 miscounted	Correct by default

Seat ticketing 1-seat waitlist • 2 freed at once	2 to 1 buyer over-allocated	Correct by default
over-allocated Postgres Two seats freed at the same moment are both offered to the same one-seat waitlister, who buys both — while the next person in line gets none. The offer step reads the waitlist without locking it. Over-allocated. Correct Convex Offering a freed seat runs in one serializable transaction that reads the next waiter's row; two seats freeing at once conflict on that read and retry, so each waiter is offered exactly one seat and no one is skipped. Correct by default.
Shop Sold-out • Many buyers	Server hangs Deadlocked	Correct by default
Deadlocked Postgres The failure branch grabs a 2nd pooled connection while still holding the 1st. With a 10-connection pool, 10 concurrent buys on a sold-out item each hold one and wait forever for another. The whole server deadlocks; /health keeps lying {ok:true}. Deadlocked. Correct Convex Convex has no connection pool to exhaust, and mutations don't hold connections across awaits, so a flood of buys on a sold-out item can't deadlock the server. Correct by default.
Warehouse 5 in stock • Many picks at once	-4 on hand oversold	Correct by default
oversold Postgres Concurrent picks race past the stock check and each decrement inventory, driving on-hand count negative while every response still returns success. Oversold. Correct Convex Each pick runs as a serializable transaction, so two picks that read the same stock can't both commit — the loser automatically retries against the updated count. There is no lock to place or forget. Correct by default.
Card game Concurrent turns	Correct	Correct by default
The Score	9 of 10 silently broken Looks right, acts wrong	100% consistent and correct Does what it is supposed to

See the exact code, app by app

The reactivity tax

Realtime, out of the box

Screens don't update themselves. On Postgres, realtime is plumbing the agent has to build by hand (SSE or WebSockets re-pushed on every write), exactly the kind of thing that gets skipped or half-shipped. On Convex there's nothing to wire: every query is a live subscription, so any committed write reaches every watching client on its own.

Convex

both screens stay live

Alice

Live

Bob:Sale is live 🚀

Bob

Live

Bob:Sale is live 🚀

Postgres

no realtime updates

Alice

In sync

Bob:Sale is live 🚀

Bob

Stale · 0s

Bob:Sale is live 🚀

The same write, two screens. Convex pushes every committed change to all watching clients instantly, no extra code. A generated Postgres app has no live channel, so the second screen just shows stale data, without updating or warning, until someone thinks to reload. Across the ten apps: Convex was live on all of them, Postgres on essentially none.

Not one model, not one run

Frontier models make the same kinds of mistakes

Complex, lower level abstractions (like Postgres) can trip up even the latest models.

GPT-5.6-Solkanban, on Postgres

Locked the column, in 5 of its 6 write paths. Missed card delete.

// POST /cards  — create
SELECT ... FROM columns ... FOR UPDATE   // locks the column
position = count(cards in column)
 
// DELETE /cards/:id  — delete
DELETE FROM cards WHERE id = $1          // no column lock
renumber the survivors 0..n-1

A concurrent create and delete corrupt the (column, position) order. The column is bricked— every new card fails from then on.

Claude Fablemarketplace, on Postgres

Locked the listing and its stock. Missed the cart.

// checkout()
SELECT ... FROM cart_items ci JOIN listings l ...
  FOR UPDATE OF l        // locks listings, not the cart
// ... decrement stock, write the order ...
 
DELETE FROM cart_items WHERE user_id = $1   // deletes the whole cart

An item added to the cart mid-checkout is swept away by that blanket delete. Never ordered, never charged, gone.

The code, the briefs, and a written breakdown of each →

Why

We made it impossible to write incorrect transactions

Every Convex mutation runs as one serializable transaction: when two writes collide, the loser automatically retries against the committed result. That isn't a best practice your agent has to remember, it's the only way a mutation runs.

function sellItem(id) {
  const item = db.get(id);
  db.set(id, { stock: item.stock - 1 });
}

Two buyers read the same item and both write it back. One sale's decrement is silently lost. Oversold.

const sellItem = mutation((ctx, id) => {
  const item = ctx.db.get(id);
  ctx.db.patch(id, { stock: item.stock - 1 });
});

The identical read-and-write, wrapped in a mutation. Convex runs them one at a time and retries the loser against the new stock. Never oversold.

Agent evals

The backend platform your agents can’t break

To test this, we asked several agents (Opus 4.8 and Fable from Anthropic, as well as GPT-5.6-Sol and GPT-5.6-Terra from OpenAI) to one-shot the same ten apps on Postgres, then again on Convex.

9 out of 10 apps silently corrupted data on Postgres. None of them did on Convex.

Zero writes lost on Convex

Database updates executed correctly, over time

More users, more problems

The apps, the load tests, and the measured data →

A verbatim example prompt

"Hey, I run a small online shop and need a backend, please.

Catalog. Each item has a name, a price, and how many are in stock.

Buy. A customer buys an item. After a purchase, the item's remaining stock goes down by one, and we keep a record of the sale.

... Build the whole backend: schema, seed a few items (some with only 1 or 2 in stock), and an API. However you think is best."

Nothing about locks, transactions, or races. Just “let people buy things”.

Don’t leave correctness to chance

On Convex, correctness is the default, not a //TODO

The real test

Correct by default

Under load, most of the Postgres apps fell over or silently corrupted data, all while answering a cheerful HTTP 200. The Convex apps had none of these issues.

Comparison of Postgres and Convex app correctness under concurrent load. Expand a failing Postgres result for details.
App under concurrent load	Postgres, as generated	Convex, correct by default
Bill splitting 2 Edits at once	Edit lost data lost	Correct by default

Booking Overlapping times • 2 requests	2 Booked double-booked	Correct by default
double-booked Postgres Two appointments whose times overlap both clear the availability check before either commits — the clash guard locks nothing when it finds no conflict — so one stylist is booked for two customers at once. Double-booked. Correct Convex Each booking runs as one serializable transaction that reads the overlapping time range; a competing appointment that overlaps conflicts on that read and retries, so a stylist can't be booked twice. Correct by default.
Kanban 10 Moves • One slot	8 Collide order corrupted	Correct by default

Ledger 10 Entry • 2 Reversals	Reversed 8x corrupted records	Correct by default

Marketplace Add item during checkout	Item lost unpaid & gone	Correct by default

Q&A forum 10 Votes at once	Score 3 miscounted	Correct by default

Seat ticketing 1-seat waitlist • 2 freed at once	2 to 1 buyer over-allocated	Correct by default
over-allocated Postgres Two seats freed at the same moment are both offered to the same one-seat waitlister, who buys both — while the next person in line gets none. The offer step reads the waitlist without locking it. Over-allocated. Correct Convex Offering a freed seat runs in one serializable transaction that reads the next waiter's row; two seats freeing at once conflict on that read and retry, so each waiter is offered exactly one seat and no one is skipped. Correct by default.
Shop Sold-out • Many buyers	Server hangs Deadlocked	Correct by default
Deadlocked Postgres The failure branch grabs a 2nd pooled connection while still holding the 1st. With a 10-connection pool, 10 concurrent buys on a sold-out item each hold one and wait forever for another. The whole server deadlocks; /health keeps lying {ok:true}. Deadlocked. Correct Convex Convex has no connection pool to exhaust, and mutations don't hold connections across awaits, so a flood of buys on a sold-out item can't deadlock the server. Correct by default.
Warehouse 5 in stock • Many picks at once	-4 on hand oversold	Correct by default
oversold Postgres Concurrent picks race past the stock check and each decrement inventory, driving on-hand count negative while every response still returns success. Oversold. Correct Convex Each pick runs as a serializable transaction, so two picks that read the same stock can't both commit — the loser automatically retries against the updated count. There is no lock to place or forget. Correct by default.
Card game Concurrent turns	Correct	Correct by default
The Score	9 of 10 silently broken Looks right, acts wrong	100% consistent and correct Does what it is supposed to

See the exact code, app by app

More features, fewer tokens

Realtime out of the box, not hand-wired

The reactivity tax

Realtime, out of the box

Convex

both screens stay live

Alice

Live

Bob:Sale is live 🚀

Bob

Live

Bob:Sale is live 🚀

Postgres

no realtime updates

Alice

In sync

Bob:Sale is live 🚀

Bob

Stale · 0s

Bob:Sale is live 🚀

Even agents make mistakes

9 of 10 Postgres apps shipped a bug it never reported

Not one model, not one run

Frontier models make the same kinds of mistakes

Complex, lower level abstractions (like Postgres) can trip up even the latest models.

GPT-5.6-Solkanban, on Postgres

Locked the column, in 5 of its 6 write paths. Missed card delete.

// POST /cards  — create
SELECT ... FROM columns ... FOR UPDATE   // locks the column
position = count(cards in column)
 
// DELETE /cards/:id  — delete
DELETE FROM cards WHERE id = $1          // no column lock
renumber the survivors 0..n-1

A concurrent create and delete corrupt the (column, position) order. The column is bricked— every new card fails from then on.

Claude Fablemarketplace, on Postgres

Locked the listing and its stock. Missed the cart.

// checkout()
SELECT ... FROM cart_items ci JOIN listings l ...
  FOR UPDATE OF l        // locks listings, not the cart
// ... decrement stock, write the order ...
 
DELETE FROM cart_items WHERE user_id = $1   // deletes the whole cart

An item added to the cart mid-checkout is swept away by that blanket delete. Never ordered, never charged, gone.

The code, the briefs, and a written breakdown of each →

Why

We made it impossible to write incorrect transactions

function sellItem(id) {
  const item = db.get(id);
  db.set(id, { stock: item.stock - 1 });
}

Two buyers read the same item and both write it back. One sale's decrement is silently lost. Oversold.

const sellItem = mutation((ctx, id) => {
  const item = ctx.db.get(id);
  ctx.db.patch(id, { stock: item.stock - 1 });
});

The identical read-and-write, wrapped in a mutation. Convex runs them one at a time and retries the loser against the new stock. Never oversold.

Want to dig into the methodology?

Models under test: Claude Opus 4.8, Claude Fable, GPT-5.6-Sol, and GPT-5.6-Terra

Review our open-sourced work

LLM Leaderboard

Curious how the models stack up?

Compare the performance of different language models on our LLM Leaderboard.

View the leaderboard

Build a production app this afternoon

Scale it for decades.

Claude Code

/plugin install convex

Codex

/plugins install convex

Works great with any model, any agent, any editor

npm create convex@latest

Read the docs Start building