Semantic Caching for AI: When "Different" Questions Can Share the Same Answer

I was at a lunch during our customer conference "AppWorld" in Las Vegas last week when the topic of LLM optimization came up. A customer across the table asked, "How does caching work for LLMs?"

My fork paused midway to my mouth as I quickly gathered my thoughts. "It's definitely a thing, and there's...uh...math...and it kind of just works..." I replied. "You're caching responses based on the meaning of queries rather than exact text matching."

Some people nodded with interest and asked for more details. As we continued the conversation, I realized how much I didn't understand how this worked... I had to dig in.

Beyond Basic Caching: The Semantic Difference

Traditional caching is straightforward: you ask for something, the system gets it, and then stores an exact copy for next time. If someone asks for the same thing again (character for character), they get the cached version. Fast, efficient, but limited. This works great in the world of HTTP, requests are pretty predictable and structured... In natural language queries, it almost seems impossible...Or is it?

Semantic caching takes this concept into higher-dimensional space—quite literally. Instead of matching exact text, it captures the meaning behind queries and matches based on semantic similarity.

The Math Behind the Magic

At the heart of semantic caching is the vector embedding. When a user submits a query to an LLM, the system converts the text into a mathematical representation, a vector in high-dimensional space. This conversion happens through an embedding model, which maps words and phrases to points in this space where semantically similar concepts cluster together.

For example, these two questions:

• "What's the capital of France?"
• "Can you tell me which city serves as France's capital?"

While textually different, their vector representations would be extremely close in the embedding space because they're asking for the same information.

The similarity between queries can be measured using mathematical techniques like cosine similarity. If two query vectors have a similarity above a certain threshold (often 0.85-0.95), they're considered to be asking essentially the same thing, and the cached response for one can be used for the other.

Embedding Models: Lightweight Yet Powerful

Yo dawg, I heard you like models so we put a model on your model...

Unlike the massive LLMs that can be hundreds of gigabytes, embedding models are relatively lightweight—typically ranging from a few hundred megabytes to about 1-2 GB.

This makes them incredibly practical for deployment. Most embedding models can run efficiently on commodity CPUs, though they benefit from GPU acceleration for higher throughput. Models like OpenAI's text-embedding-ada-002, BERT, or Sentence-BERT variants can process hundreds or thousands of queries per second on standard hardware.

For context, while a full-featured LLM like GPT-4 requires significant computational resources, you could run a production-grade embedding model on a moderately-sized virtual machine with 4-8 CPU cores and 16GB of RAM.

Gateway Implementation: Practical and Powerful

"Could we set up our own caching layer between our apps and the commercial LLM APIs?" another customer at the table asked.

"I don't see why not," I replied. "We have customers doing that today in some cases."

A semantic caching gateway is feasible as an intermediary between your applications and commercial LLM APIs like OpenAI, Anthropic, etc. The architecture would look something like this:

1. Your application sends a query to the gateway
2. The gateway converts the query to a vector using a local embedding model
3. It checks a vector database for similar cached queries
4. If found, it returns the cached answer immediately
5. If not, it forwards the request to the commercial LLM API, caches the response, and returns it

This approach gives you the best of both worlds: the power of state-of-the-art LLMs without repeatedly paying for identical or nearly identical queries.

How It Works in Practice

Here's a simplified workflow:

1. User submits a query: "How do I make chocolate chip cookies?"
2. The system converts this into a vector using an embedding model
3. The system checks if any existing cached vectors are similar enough
4. If a match is found, return the cached response
5. If no match, send to the LLM, cache the response and its vector

A week later, when someone asks "What's a good recipe for cookies with chocolate chips?", the system can identify it as semantically similar and serve the cached response—saving processing time and reducing API costs.

Real-World Examples

Let's look at some pairs of queries that could benefit from semantic caching:

Example 1: Technical Support

• Original query: "My laptop won't turn on after installing the latest update."
• Similar query: "After updating my computer, it doesn't boot anymore."

Example 2: Product Information

• Original query: "What are the dimensions of the Model X smartphone?"
• Similar query: "How big is the Model X phone?"

Example 3: Legal Advice

• Original query: "What are the requirements for establishing copyright protection?"
• Similar query: "How do I ensure my work is protected by copyright law?"

In each case, generating a fresh response would be redundant and costly, yet a traditional cache would miss these opportunities.

The ROI of Semantic Caching: GPT-4.5 Case Study

At some point in our lunch, the conversation shifted to the newly released GPT-4.5 model, designed specifically for creative tasks and agentic planning.

"Have you seen the pricing? Input tokens are $75 per million, cached input tokens are $37.50 per million, and output tokens are $150 per million."

Let's break down the potential savings with a realistic scenario:

Imagine an enterprise application processing 10,000 queries daily, with an average of 1,000 tokens per input and 2,000 tokens per output response. Without any caching:

• Daily input cost: 10,000 queries × 1,000 tokens × $75/1M tokens = $750
• Daily output cost: 10,000 queries × 2,000 tokens × $150/1M tokens = $3,000
• Total daily cost: $3,750
• Monthly cost: $112,500

Now, let's assume a semantic caching system that achieves a 40% cache hit rate (quite achievable for many applications). With OpenAI's cached input pricing and avoiding unnecessary output generation:

• Daily uncached input cost (60% of queries): 6,000 queries × 1,000 tokens × $75/1M tokens = $450
• Daily cached input cost (40% of queries): 4,000 queries × 1,000 tokens × $37.50/1M tokens = $150
• Daily uncached output cost (60% of queries): 6,000 queries × 2,000 tokens × $150/1M tokens = $1,800
• Daily cached output cost (40% of queries): $0 (reusing previous outputs)
• Total daily cost: $2,400
• Monthly cost: $72,000

That's a monthly saving of $40,500—or almost $500,000 annually—just by implementing semantic caching. And this doesn't even account for the reduced latency users experience when receiving cached responses.

Implementation Considerations

Implementing semantic caching isn't without challenges:

1. Threshold setting: Too strict, and you miss caching opportunities; too loose, and you serve inappropriate responses.
2. Cache invalidation: Information can become outdated, requiring mechanisms to expire or refresh cached content.
3. Vector database: You'll need an efficient way to store and query high-dimensional vectors—solutions like Pinecone, Weaviate, or Qdrant are often used.
4. Embedding model choice: The quality of your semantic matches depends heavily on your embedding model's performance.
5. Response variability: Some queries may benefit from fresh, varied responses rather than cached ones.

The ROI of Semantic Caching

For high-volume AI applications, the savings can be substantial. Consider a system handling 1 million queries per day. If 30% of those can be served from a semantic cache, and each LLM call costs $0.01, that's a daily saving of $3,000—or over $1 million annually.

Beyond cost, there are performance benefits. Cached responses can be delivered in milliseconds, while generating new responses might take seconds.

Conclusion

Whether you're running a customer support chatbot or an internal knowledge base, semantic caching represents one of the most effective ways to optimize your LLM operations. And with embedding models being lightweight enough to run on standard hardware, implementing a caching gateway between your applications and commercial LLM APIs is not just theoretical—it's a practical solution that organizations can deploy today to significantly reduce costs while maintaining performance.

Shameless Plug

We recently announced the GA of our AI Gateway which, among other things, provides the ability to do Semantic Caching. You should check it out.