Implements Apple's QuiCK paper as a Convex component, providing FIFO and priority queue processing with retries and concurrent execution.
npm install @danthegoodman/quick-convexA [QuiCK](https://www.foundationdb.org/files/QuiCK.pdf)-style queue implementation as a Convex component.
## Queue behavior
- Supports `"vesting"` and `"fifo"` order modes.
- Uses pointer-based scanning and leasing for concurrent processing.
- `managerSlots` controls maximum concurrently running managers (default `10`).
- `workersPerManager` controls how many items a manager dequeues per `queueId` pass (default `10`).
- Includes cron-based recovery and pointer garbage collection.
### Choosing an ordering mode
- Use `"vesting"` when throughput is the priority. Ready items can run as soon as they are due, so delayed/retried items do not block newer ready work in the same queue.
- In `"vesting"` mode, ready items are dequeued by `priority` first, then by `vestingTime` within that priority tier.
- Cross-queue priority is best-effort rather than strict because queue scheduling is still pointer-based.
- Use `"fifo"` when strict per-`queueId` ordering is required. This enforces head-of-line semantics for that ordering domain. In `"fifo"` mode, a delayed/retrying head item stalls the rest of that same `queueId` until it is ready again.
In practice, FIFO queues are often a cleaner and more performant alternative to creating many `maxParallelism: 1` workpools (one per ordering domain). With Quick FIFO, use `queueId` as the domain key (for example a user id, account id, or aggregate id), and each domain (`queueId` value) stays ordered while different domains can still process in parallel.
## Compare to Convex Workpools
Quick is heavily inspired by Convex Workpools and the QuiCK paper. Workpools are excellent and production-proven, and this component builds on many of the same ideas.
### Workpools strengths
- Slightly lighter weight runtime model.
- Officially maintained by the Convex team.
- Operationally simpler in many common setups.
- Production-proven at scale in Convex.
### Workpools edge case to be aware of
- In some bursty scale-up patterns (idle `0` to many scheduled items), contention can appear around work claiming.
- Today, Workpools do not retry onComplete timeout failures.
### Quick strengths
- Multiple ordering modes, especially strict per-domain FIFO via `queueId`.
- FIFO is much easier to model than emulating ordering via many `maxParallelism: 1` workpools.
- Faster and lower contention ramp from idle to heavy load.
- OnComplete timeout failures are retried up to 2 times.
- QuiCK model proven at scale at Apple (but not this implementation!)
### Tradeoff to keep in mind
- Quick is a bit heavier per unit work when load is low or not amortized (more queue-management actions/mutations around each job).
## Quick Class API
Use the class API for enqueueing work:
```ts
import { Quick } from "@danthegoodman/quick-convex";
import { components, api } from "./_generated/api";
const quickVesting = new Quick(components.quickVesting, {
defaultOrderBy: "vesting",
workersPerManager: 25,
retryByDefault: true,
defaultRetryBehavior: {
maxAttempts: 5,
initialBackoffMs: 250,
base: 2,
},
});
const quickFifo = new Quick(components.quickFifo, {
defaultOrderBy: "fifo",
});
```
### Worker function contract
Workers must accept this argument shape:
```ts
{
payload: TPayload;
queueId: string;
}
```
This applies to both action and mutation workers.
### Enqueue action worker
```ts
export const enqueueEmail = mutation({
args: { userId: v.string() },
handler: async (ctx, args) => {
return await quickVesting.enqueueAction(ctx, {
queueId: args.userId,
priority: 12,
fn: api.jobs.sendEmailWorker,
args: { userId: args.userId },
runAfter: 5_000,
});
},
});
```
### Enqueue mutation worker
```ts
export const enqueueMutationWorker = mutation({
args: { userId: v.string() },
handler: async (ctx, args) => {
return await quickVesting.enqueueMutation(ctx, {
queueId: args.userId,
fn: api.jobs.processUserMutationWorker,
args: { userId: args.userId },
});
},
});
```
### Batch enqueue (multi-function)
`enqueueBatchAction` and `enqueueBatchMutation` accept per-item function refs and dedupe handle creation per unique function in the batch.
```ts
export const enqueueBatch = mutation({
args: { queueId: v.string() },
handler: async (ctx, args) => {
return await quickFifo.enqueueBatchAction(ctx, [
{
queueId: args.queueId,
fn: api.jobs.workerA,
args: { value: 1 },
},
{
queueId: args.queueId,
fn: api.jobs.workerA,
args: { value: 2 },
},
{
queueId: args.queueId,
fn: api.jobs.workerB,
args: { value: 3 },
},
]);
},
});
```
### Retried onComplete callback
`onComplete` is always a mutation handle and runs for both action and mutation workers.
Quick persists completion state (`phase: "onComplete"`) and resumes there after crashes, so completion handlers are retried safely up to 2 times.
```ts
import { vOnCompleteArgs } from "@danthegoodman/quick-convex";
export const onEmailComplete = mutation({
args: vOnCompleteArgs(v.object({ userId: v.string() })),
handler: async (_ctx, args) => {
// args: { workId, context, status, result }
return null;
},
});
export const enqueueEmail = mutation({
args: { userId: v.string() },
handler: async (ctx, args) => {
return await quickVesting.enqueueAction(ctx, {
queueId: args.userId,
fn: api.jobs.sendEmailWorker,
args: { userId: args.userId },
onComplete: {
fn: api.jobs.onEmailComplete,
context: { userId: args.userId },
},
});
},
});
```
`vOnCompleteArgs()` without a context validator uses `context: any`.
### Retry configuration
- Set class defaults in `new Quick(component, { retryByDefault, defaultRetryBehavior })`.
- Override per item with `retry`:
- `retry: false` disables retries for that item.
- `retry: true` uses class/default retry behavior.
- `retry: { maxAttempts, initialBackoffMs, base }` sets per-item behavior.
### Priority in vesting mode
- `priority` is optional on `enqueueAction`, `enqueueMutation`, and batch items.
- Missing `priority` defaults to `0`.
- Supported values are integers in `0..15`.
- Higher numbers are higher priority.
- Priority only affects `"vesting"` mode.
quick-convex provides strict FIFO ordering per queueId, ensuring tasks for each user are processed sequentially while different users can process in parallel. Use the FIFO mode with userId as the queueId to maintain ordering within each user's task queue.
quick-convex includes built-in retry mechanisms with exponential backoff and configurable max attempts. Set retry behavior at the class level or override per job, and use onComplete callbacks that are automatically retried up to 2 times on timeout failures.
The vesting mode in quick-convex supports priority levels 0-15 where higher numbers run first. Ready items are dequeued by priority, then by vesting time, allowing high-priority work to jump ahead while maintaining fairness within priority tiers.
Use enqueueBatchAction or enqueueBatchMutation to submit multiple jobs at once with different worker functions. The component automatically deduplicates handle creation per unique function in the batch for efficiency.
FIFO mode enforces strict ordering per queueId where delayed items block subsequent items in the same queue. Vesting mode prioritizes throughput, allowing ready items to run immediately even if earlier items are delayed or retrying. Use FIFO for strict ordering requirements and vesting for maximum throughput.
quick-convex offers multiple ordering modes including strict FIFO per domain, while Workpools are lighter weight and officially maintained. quick-convex handles scale-up patterns better and retries onComplete failures, but has more overhead per job at low volumes. Both are production-ready solutions.
Yes, quick-convex supports retry configuration at multiple levels. Set defaults when creating the Quick instance, then override per job with retry: false, retry: true, or retry: { maxAttempts, initialBackoffMs, base } for custom behavior.
Use the onComplete callback which runs as a mutation after both successful and failed jobs. quick-convex persists completion state and retries the onComplete handler up to 2 times on failures, ensuring reliable cleanup and post-processing logic.