Batch Processor

Efficiently process large datasets in Convex with batch accumulator and table iterator utilities for optimized performance.

Installation

npm install convex-batch-processor

About Batch Processor

Batch processing component for Convex - batch accumulator and table iterator

Benefits

Process large datasets without hitting Convex query time limits
Accumulate batched operations with built-in error handling and retry logic
Iterate through tables efficiently with automatic pagination and memory management
Reduce function execution costs by batching operations instead of individual calls

Use cases

how to process large tables in Convex without timeout

The convex-batch-processor provides a table iterator that automatically handles pagination and prevents query timeouts. It processes records in configurable batch sizes, allowing you to iterate through entire tables while staying within Convex execution limits.

batch operations in Convex mutations

Use the batch accumulator to collect multiple operations and execute them efficiently. The component handles batching logic, error recovery, and ensures optimal performance when processing multiple records or performing bulk updates in Convex.

Convex bulk data processing patterns

The batch processor implements proven patterns for handling large datasets in Convex. It combines table iteration with batch accumulation to process thousands of records efficiently while maintaining data consistency and handling edge cases.

Frequently asked questions

How does convex-batch-processor handle Convex query limits?

The convex-batch-processor automatically manages batch sizes and execution time to stay within Convex limits. It uses pagination for table iteration and configurable batch sizes for operations, ensuring your functions don't timeout while processing large datasets.

Can I customize batch sizes with this component?

Yes, convex-batch-processor allows you to configure batch sizes based on your specific needs. You can set different batch sizes for table iteration and operation accumulation, optimizing performance for your data patterns and processing requirements.

Does this component handle errors during batch processing?

The convex-batch-processor includes built-in error handling and retry logic for failed operations. It can continue processing remaining batches when individual operations fail, and provides detailed error reporting to help you handle edge cases in your Convex functions.

Is convex-batch-processor compatible with all Convex table types?

Yes, convex-batch-processor works with any Convex table structure. The table iterator is generic and adapts to your schema, while the batch accumulator can handle any type of operation or data transformation you need to perform on your records.