Cache Hits vs. Cache Misses in LLMs: What Happens on the GPU During Inference

Have you ever wondered what does it mean by Cache Hit and Cache Miss in most of AI model platform now? When you ask a model like Claude, ChatGPT, or DeepSeek a question, wait for an answer, and then ask for a follow-up, it feels completely natural. You talk to it just like a human assistant referring back to ideas you mentioned before, without having to re-explain yourself every time. To you, it feels like a seamless, continuous stream of thought. But behind the curtain, something very different is happening.

AI models don't actually have a human-like memory. They don't "remember" your previous message the way you remember about yourself. In fact, every single time you hit Send, the AI is handed the entire conversation from the very beginning and forced to read all of it all over again just to understand your latest question.

Imagine having to re-read an entire book every time someone asks you a question about Chapter 10. If an AI system did that from scratch on every turn, it would require a huge amounts of computing power, create delays, and burn through expensive hardware resources.

So, how do AI systems manage to process massive amounts of past context instantly without bringing their supercomputers to a crawl or blowing up the monthly bill?

It all comes down to a fundamental concept borrowed from traditional computing, now adapted for the modern AI era: Cache Misses and Cache Hits.

What is a "KV Cache”

To understand how a "hit" or a "miss" actually happens, we first need to look at what the AI is doing when it "reads" your text. When you type a sentence into an AI model, it doesn't read words the way humans do. Instead, the AI breaks your message into tiny pieces called tokens (words or parts of words) and translates them into complex mathematical coordinates.

During this translation step, the AI’s graphics processors (GPUs) do heavy mathematical calculations to create two critical pieces of digital data for every single token:

• Keys (K): Think of these as index tabs. They help the model categorize what each word is and how it connects to other words.
• Values (V): Think of these as the actual information stored inside those index, the raw meaning of the text.

If the AI threw those KV calculations away after every single message, it would have to redo that heavy math for your entire past chat history every time you sent a reply. Its expensive! Btw, please pardon Gemini for that misspell in the diagram above😅.

This is where the KV Cache comes in.

The KV cache is simply a high-speed temporary memory bank sitting directly on the GPU. Once the AI calculates the Keys and Values for your message, it saves those mathematical results in this cache.

For the techies

In Transformer-based Large Language Models, the model processes input text by passing token representations through multiple Attention Layers. Inside each attention head, the model converts input token embeddings into three separate vectors via learned projection matrices ($W_Q$, $W_K$, $W_V$):

1. Queries (Q): Represents the current token attempting to look back and absorb context from preceding tokens.
2. Keys (K): Represents the label or address vector for each past token, encoding what kind of information that token contains and where it stands in the sequence.
3. Values (V): Represents the content or feature representation vector of that past token the actual contextual information that will be passed forward.

How Attention Uses K and V

When generating a new token, the Transformer computes the attention weights by taking the dot product of the current token’s Query (Q) vector with the Key (K) vectors of all preceding tokens:

Attention(Q, K, V) = softmax((Q * K^T) / sqrt(d_k)) * V

• The QK^T dot product calculates a score indicating how relevant every previous token is to the current token.
• Softmax normalizes these scores into probabilities (attention weights).
• The attention weights are then multiplied by the Value (V) vectors, aggregating the contextual information needed to predict the next token.

So, what is Cache Hit and Cache Miss?

When you send a prompt to an LLM, the system checks whether the early tokens in your prompt match any K and V vectors already stored in GPU memory from a previous request. This leads to two possible outcomes:

Cache Hit:

• What happens: The model recognizes that the initial tokens (the "prefix") match K and V vectors already sitting in VRAM. It skips recalculating those tokens entirely and reuses the cached states.
• The Result: Near-instant processing speed and a massive cost reduction, typically 98%–99% cheaper on input token billing, as you avoid paying for GPU matrix multiplication.

Cache Miss:

• What happens: The prompt is brand-new, or the previous cache was cleared/evicted from GPU memory. The GPU must process every token from scratch to compute its K and V vectors.
• The Result: High latency (time-to-first-token) and full price for input tokens.

Real example:

Let say you are chatting with ChatGPT, for a certain topic.

1. You: "Tell me about cat. I love to hear more about cat especially Persian.”
2. ChatGPT: “<ChatGpt replies>”
3. You: “Now tell me about cat's house.””
4. ChatGPT: “<ChatGpt replies>”
5. You: “But wait, does that cat's house suitable for my cat?”
6. ChatGPT: “Yes, the house is suitable for Persian cat. <ChatGPT replies>”

What actually happens are:

1. Turn 1 (First Chat)
   • You: "Tell me about cat. I love to hear more about cat especially Persian."
   • What happens: This is brand-new information. The GPU must process every word from scratch.
   • Billing: 100% Cache Miss (charged full standard rate).
2. Turn 2 (Second Chat)
   • You: "Now tell me about cat's house."
   • What happens: The system sends Turn 1 + Turn 2 together so the model remembers the conversation.
     • Turn 1 (Past chat) → Cache Hit (loaded instantly from KV cache in temp memory).
     • Turn 2 (New question) → Cache Miss (processed by GPU).
   • Billing: You pay ~98% less for Turn 1, and full price for the small new line.
3. Turn 3 (Third Chat)
   • You: "But wait, does that cat's house suitable for my cat?"
   • What happens: The AI responds "Yes, Persian cat is suitable..." because it pulled the "Persian" context right out of the KV cache from Turn 1!
     • Turns 1 & 2 (Past context) → Cache Hit (retrieved from memory).
     • Turn 3 (New question) → Cache Miss (processed by GPU).
   • Billing: Huge Cache Hit for the long past history + small Cache Miss for Turn 3.

Thats it. Thats how to understand Cache Hit and Cache Miss from AI model providers’ perspective. Now you understand why Cache Hit cost around ~98% cheaper than Cache Miss.

Is Cache Hit and Cache Miss similar to Memory

The answer is no. KV cache is similar in terms of its usage to ready up the model with the existing context, but fundamentally different. It forgets everything when the session ends. When you close a chat window or finish an API call, the GPU releases that VRAM allocation. The KV Cache vanishes. A memory system ie episodic memory, by contrast, must persist over days, weeks, or years.

Asides, it operates strictly within the active context window. The KV Cache simply prevents the GPU from having to re-calculate past tokens within the current prompt. It is an engineering optimization for computation speed during an active context, not an archive of past experiences.

So yeah, that is for KV cache understanding. And thats the importance of a good prompting. By understanding how KV caching works under the hood, you can understand the importance of a good prompts to maximize prefix reuse, getting lightning-fast responses and save a huge bucks without sacrificing a single drop of context.

Want to read more about prompting, you can read it here.

Article: https://arifmustaffa.com/blog/2dedd414-9dba-80d0-a47c-d74bb5784caa

Happy KV-ing😄.