Serving an LLM is often described as a capacity problem: add replicas, watch queues, keep GPUs busy. That framing misses a useful fact about many real workloads. Two requests may ask the same model for different answers while sharing a large amount of already processed context: a system prompt, a repository snapshot, a conversation history, or the stable front half of a retrieval-augmented prompt.

A model has to process that context before it can generate its first new token. If one worker has already done that work, sending the next similar request to another idle worker can be an expensive kind of fairness. The new worker starts from zero; the first worker has useful state but may receive no work.

That trade-off is the point of Cache-aware scheduling. It is an approach to placing LLM requests that considers reusable execution state alongside ordinary signals such as queue depth, available capacity, and latency targets. In current LLM systems, the most important reusable state is usually the KV cache: stored attention data from tokens the model has already processed.

Plain English: A free GPU is not automatically the fastest GPU for a request. The fastest one may be the GPU that has already read most of that request’s context.

The routing problem hidden inside prefix caching

A transformer does not reread a prompt in the everyday sense, but it does compute internal key and value representations for every input token as it builds the context used for attention. A KV cache retains those representations. When a later request has the same compatible beginning, the serving system can reuse the cached work and compute only the unmatched remainder.

This is Prefix caching: retaining and reusing computation for a shared token prefix. It can reduce the prefill work—the phase in which the model processes input context before generation begins. Prefill is often a major contributor to time to first token, especially for long prompts.

But a cache is only useful where it is available. In a replicated service, each worker may hold a different cache contents. A cache-blind load balancer can spread requests evenly and still cause every replica to recompute the same large prefix. Cache-aware scheduling adds a question before dispatch: which worker, or which cache tier, is most likely to have compatible state already resident?

Amazon’s recent SageMaker feature makes this idea concrete. Its prefix-aware routing sends requests with a shared prefix to the same inference instance, and exposes controls for prefix length and a concurrency threshold. The API documentation describes those routing controls. AWS reports up to 77% lower P50 time-to-first-token and higher KV-cache hit rates in its stated benchmark; that is a vendor result for that workload and configuration, not a universal saving. The benchmark details and qualification are AWS’s.

Why does this matter now? LLM applications are increasingly built around deliberately stable context. A coding assistant may prepend repository instructions and tool definitions. A support agent may carry a conversation and policy prompt over multiple turns. A RAG application may put tenant-specific instructions, retrieval policy, and recurring source material before the user’s question. As contexts grow, placement becomes a serving architecture decision rather than a minor load-balancer setting.

How the decision works

The exact implementation differs, and the terminology is not fully standardized. “Prefix-aware routing” commonly means choosing a replica based on a shared prefix. “Cache-aware” is broader: it can cover routing across replicas, choosing between GPU, CPU, or storage tiers, managing cache migration, and coordinating separate prefill and generation workers.

A useful mental model has five steps.

First, the system establishes a cache-relevant identity for an incoming request. This may be derived from tokenized prompt prefixes or from metadata that predicts a matching prefix. Token equality alone is not necessarily enough: model version, tokenizer configuration, adapters, hidden system inputs, and authorization context can change whether reuse is correct.

Second, the control plane estimates cache residency. It needs to know which worker has matching blocks, how much of the prefix matches, and whether those blocks are still likely to remain available. Implementations can organize cache entries in blocks or in a prefix-oriented structure such as a radix tree. The important architectural fact is not the data structure; it is that cache location becomes scheduling input.

Third, the scheduler scores candidates. A longer reusable prefix may avoid more prefill computation, but a worker with the best match might also have a deep queue. The policy therefore weighs expected reuse against queueing, capacity, and a latency objective. Sending every matching request to one replica can turn a warm cache into a hotspot.

Plain English: The scheduler is making a trade: reuse old work on a busy machine, or redo that work on a less busy one.

Fourth, the selected worker looks up and reuses matching KV blocks, then processes the suffix that differs. A cache hit is not a free request. The model still needs to process new tokens and generate an answer; long output generation can dominate the total time even when prefill is cheap.

Finally, the system retains, evicts, or moves state according to memory pressure and predicted future value. Cache lifetime is part of the scheduling policy. A route based on yesterday’s cache map fails if a worker has restarted, a long generation has displaced the relevant blocks, or a cache index has become stale.

The same logic becomes more complicated with Disaggregated serving, an architecture that separates prefill from decode—the token-by-token generation phase—often into different worker pools. Cache-aware placement can choose a prefill worker with the matching state and then make that state available to a decode worker. That may improve utilization, but transferring or exposing state adds cost and coordination. It is not the same thing as cache-aware scheduling; it is an architecture in which cache-aware scheduling has more places to act.

What it is not

Cache-aware scheduling is not ordinary load balancing with a new label. Round-robin, least-connections, and utilization-based policies answer “where is capacity?” Cache-aware policies also ask “where is prior computation?” Neither question should automatically win.

It is also not prefix caching itself. Prefix caching is the mechanism that stores and reuses state. A single-worker server can have excellent prefix caching without any routing decision at all. Conversely, a router can preserve session affinity without knowing whether a compatible cache entry survived. The combined system needs both reuse and informed placement.

Nor is it Model routing. Model routing selects a model based on capability, cost, or quality. Cache-aware scheduling usually starts after that choice: it places work for the already chosen model. Mixing the two decisions can be sensible, but they optimize different things.

Finally, it is not a license to treat equal tokens as equal security contexts. A Cache namespace separates cache entries that must not be shared. A service handling multiple tenants or permission levels needs identity or salting rules that include relevant tenant and authorization context. Otherwise, a superficially identical prefix can produce unsafe reuse or incorrect semantics.

Example: a repository assistant under real traffic

Consider a hypothetical internal coding assistant. Every request includes a stable system prompt, tool definitions, organization policy, and a compact repository map. The user’s question and selected files follow that shared beginning. Several engineers may ask about the same repository within a short period.

With cache-blind balancing, request one lands on replica A and builds the prefix state. Request two, with nearly the same context, lands on replica B because B is less busy. B repeats the expensive prefill. Under a cache-aware policy, the router recognizes a compatible prefix and compares the expected reuse on A against A’s queue penalty. If A is only modestly busier, the policy sends request two to A. If A is saturated, it can send the request to B rather than protecting locality at the expense of an unacceptable wait.

The prefix identity must be stricter than “same repository.” A different model release, repository revision, policy version, adapter, or access scope may make reuse invalid. The service should partition its cache accordingly. It should also measure the result at the request level: prefix length, estimated and realized reused tokens, cache-hit rate, queue time, time to first token, end-to-end latency, eviction rate, and the worker selected.

Those measurements are more useful than a single aggregate hit rate. A high hit rate may coexist with worse tail latency if one warm replica receives too much traffic. Likewise, lower average prefill time may be irrelevant if the application is dominated by long generation. The service needs an SLO—a defined reliability or latency target—and should judge the policy against representative traffic rather than a tidy sequence of identical prompts.

Where it breaks

The first failure mode is weak reuse. Highly variable prompts, short common prefixes, and entries evicted before a related request arrives leave little work to save. A burst of identical requests can also arrive before the first request has finished populating the cache.

The second is imbalance. Locality is a concentrating force. It can leave cache-bearing replicas overloaded while cold replicas sit underused. Workloads with popular documents, shared agent instructions, or a few active repositories are especially prone to this. Cache-aware routing therefore needs explicit queue or concurrency limits, fallbacks, and a willingness to sacrifice a hit when the latency cost is too high. Additional preprocessing and cache tracking, as well as load-imbalance trade-offs, are documented concerns in cache-aware designs. NVIDIA discusses the tracking and placement trade-offs; BanaServe studies balancing cache state in disaggregated serving. The BanaServe paper is a separate research source.

The third is control-plane correctness. A stale view of cache residency, worker churn, cache-version mismatch, or an incorrect compatibility key can quietly turn an optimization into a latency regression or a correctness problem. Treat cache metadata as distributed state: define what happens after a restart, how entries are invalidated, and which component is authoritative.

What a senior engineer can do

Start by finding stable prefixes in production traces. Do not infer reuse potential from prompt templates alone; measure token-level commonality across actual requests, arrival timing, and cache lifetimes. Segment by workload: interactive chat, RAG, coding, and batch processing may behave very differently.

Then establish a cache-aware baseline. For a representative traffic replay, compare cache-blind placement with an affinity policy that includes a load penalty. Track P50 and tail time to first token, end-to-end latency, queue time, cache hits, reused-token estimates, eviction, and per-replica utilization. Preserve the model, hardware, traffic shape, and SLO when comparing results.

Make compatibility and isolation explicit. Version cache identity with model and tokenizer changes; include adapter and relevant hidden-context versions; partition by tenant and authorization scope. Test negative cases deliberately: two prompts that look alike but must not share state should never map to the same reusable entry.

Finally, instrument the decision, not just the outcome. OpenTelemetry’s GenAI semantic conventions define fields and operation names for agents, conversations, workflows, providers, tools, retrieval, and cached-token usage. The specification also warns that tool-call arguments and results can be sensitive. The practical goal is modest: when latency changes, you should be able to tell whether the cause was a missed prefix, an eviction, a queueing choice, a transfer, or the model’s generation work.

Cache-aware scheduling does not make context free. It makes a previously hidden resource—where already-computed context lives—visible to the system that decides where work goes. For serving teams, that is the useful shift: stop treating routing and caching as separate optimizations, and evaluate them as one policy with latency, capacity, fairness, correctness, and isolation consequences.