LLM Cache

Cache LLM API requests and responses in Convex with tiered TTL, time travel debugging, and request normalization to reduce costs and latency.

Installation

npm install @mzedstudio/llm-cache

About LLM Cache

LLM request/response caching with tiered TTL, time travel, and request normalization for Convex.

Benefits

Reduce LLM API costs by caching identical or similar requests with intelligent normalization
Improve application performance with configurable TTL caching layers for fast response times
Debug LLM interactions using time travel features to inspect cached request/response pairs
Scale LLM-powered features without hitting rate limits through efficient cache management

Use cases

how to cache OpenAI API responses in Convex

The @mzedstudio/llm-cache component provides built-in caching for LLM API responses in your Convex backend. It automatically stores request/response pairs with configurable TTL and handles cache invalidation, reducing redundant API calls to OpenAI or other LLM providers.

reduce LLM API costs with request caching

This component implements intelligent request normalization and tiered caching to minimize duplicate LLM API calls. It can identify similar requests and serve cached responses, significantly reducing your API usage costs while maintaining response quality.

debug LLM requests with time travel

The time travel feature lets you inspect historical LLM request/response pairs stored in your Convex database. This enables debugging of LLM interactions, A/B testing different prompts, and analyzing response patterns over time.

Frequently asked questions

How does request normalization work in LLM caching?

The @mzedstudio/llm-cache component normalizes LLM requests by standardizing parameters like whitespace, capitalization, and prompt formatting before checking the cache. This ensures that functionally identical requests hit the cache even with minor formatting differences, maximizing cache efficiency.

What is tiered TTL in LLM caching?

Tiered TTL allows you to set different expiration times for different types of cached LLM responses. For example, you might cache static prompt responses for 24 hours but dynamic user-specific responses for only 1 hour, optimizing both performance and data freshness.

Can I use this component with any LLM provider?

Yes, @mzedstudio/llm-cache is provider-agnostic and works with OpenAI, Anthropic, Google, or any other LLM API. It caches the request/response data structure regardless of the underlying LLM service, making it flexible for multi-provider setups.

How does time travel debugging work for LLM requests?

The time travel feature stores all cached LLM interactions with timestamps in your Convex database. You can query historical request/response pairs, replay specific interactions, and analyze how your LLM responses have changed over time, making it easier to debug and optimize prompts.