Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

OasisKV: Scaling In-Decode KV Cache Beyond HBM with Lookahead Sparse Prefetching

arXiv:2608.080972026-08-07

OasisKV keeps most of an LLM's key-value memory off the GPU and prefetches just the right pieces ahead of time, nearly doubling decoding throughput

When a large language model generates text token by token, it must keep referring back to a growing key-value (KV) cache of past tokens, and this cache quickly fills up the GPU's fast but limited HBM memory as contexts grow. OasisKV reuses the 'draft tokens' already produced by speculative decoding to accurately predict, one step ahead, which KV blocks the next decoding step will actually need, then prefetches them in the background from CPU or remote memory into GPU HBM. Built on top of vLLM, it keeps accuracy within 0.7 points of full attention while raising decode throughput by up to roughly 2x.

METAL MEDIA explanatory visual

How OasisKV predicts and prefetches KV blocks ahead of decoding

Evidence statusMeasured results reported

  1. The problemLong-context decoding requires keeping a growing KV cache resident in GPU HBM, which caps batch size and throughput
  2. Lookahead predictionDraft tokens from speculative decoding are used to rank which KV blocks the next decode step will need, without training a dedicated predictor
  3. Selection and capped transferPredicted blocks are compared against resident GPU blocks; only missing ones are fetched, capped per step so transfer fits within the PCIe bandwidth budget
  4. Asynchronous background pipelinePrediction, selection, and transfer run as overlapped background stages across layers, hiding data movement behind ongoing GPU computation
  5. OutcomeBuilt on vLLM, throughput improved up to about 2.1x with accuracy loss under 0.7 points, and about 2x under prefill-decode disaggregated serving
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. Problem: as agentic workloads push context lengths far beyond chatbot-era prompts, the KV cache dominates GPU high-bandwidth memory (HBM) usage, capping batch size and throughput.
  2. Method: OasisKV reuses the draft tokens already generated by speculative decoding as a training-free signal to predict which KV blocks the next decode step will need, then runs a background pipeline (predict, select, transfer) that prefetches those blocks from host or remote memory into HBM before they're needed.
  3. Implementation: prototyped on vLLM with a head-wise logical-to-logical mapping over the paged KV pool and a capped-eviction policy that bounds how many KV blocks are fetched per step so PCIe transfer stays hidden behind compute.
  4. Result: under a 2,048-token KV budget, accuracy stays within 0.7 points of full attention (which keeps the entire KV cache), while throughput rises 1.69x over dense vLLM on a reasoning workload (at 0.1 points accuracy loss) and up to 2.1x on multi-GPU long-context serving.
  5. Extension: under prefill-decode disaggregated serving, OasisKV reaches about 2x dense throughput while admitting each request with 6.5-9.7x less KV data and using 2.2-2.6x less decode-node host memory than transferring the full KV cache.
Figure 1. KV-cache memory roofline. Decode throughput is capped by Token-KV intensity × KV-cache bandwidth (three-tier slope-1 roofs) and by the dense/sparse throughput ceilings. Higher Token-KV intensity means more KV reuse (e.g., GQA (1), MLA (5)) or more KV sparsity (e.g., NSA (42), DSA (4)). It is defined as the tokens generated per decode pass per KV-cache byte transferred over HBM or off-GPU IOs. Two throughput ceilings are determined by the maximum batch size and minimum time per output token (TPOT) bounded by compute and weight-loading time.
Figure 1. KV-cache memory roofline. Decode throughput is capped by Token-KV intensity × KV-cache bandwidth (three-tier slope-1 roofs) and by the dense/sparse throughput ceilings. Higher Token-KV intensity means more KV reuse (e.g., GQA (1), MLA (5)) or more KV sparsity (e.g., NSA (42), DSA (4)). It is defined as the tokens generated per decode pass per KV-cache byte transferred over HBM or off-GPU IOs. Two throughput ceilings are determined by the maximum batch size and minimum time per output token (TPOT) bounded by compute and weight-loading time.
Figure 2. The per-output-token decode latency (TPOT) breakdown of on-demand KV retrieval over PCIe (Qwen3-32B, one H100 SXM-HBM3 GPU, context length 32K, BF16). All latencies are computed from a roofline model, assuming 10% of the attended KV is fetched from CPU DRAM each step. The percentage above each bar denotes the KV retrieval overhead.
Figure 2. The per-output-token decode latency (TPOT) breakdown of on-demand KV retrieval over PCIe (Qwen3-32B, one H100 SXM-HBM3 GPU, context length 32K, BF16). All latencies are computed from a roofline model, assuming 10% of the attended KV is fetched from CPU DRAM each step. The percentage above each bar denotes the KV retrieval overhead.
Table 1. Accuracy under the same 2,048-token KV budget. Each retrieval method is read against the full-attention anchor of its own stack: Δ is the difference from that stack’s Full column (Quest, FreeKV vs. Full (HF); Ours vs. Full (vLLM)), never across stacks.
HuggingFace Transformers stackvLLM stack
Dataset / SubsetMetricFullQuestΔFreeKVΔFullOursΔ
Long input — Llama-3.1-8B-Instruct, LongBench v2
Overall29.6229.42−0.2029.03−0.5930.2329.62−0.61
Short34.4434.440.0035.00+0.5635.5633.89−1.67
Medium28.3728.84+0.4726.51−1.8627.4427.440.00
Long24.0722.22−1.8524.070.0026.8526.850.00
Long input — Qwen3-8B, LongBench v2
Overall32.2131.61−0.6031.01−1.2033.6033.20−0.40
Short36.6737.22+0.5536.670.0039.4438.33−1.11
Medium29.3026.98−2.3227.44−1.8629.3030.70+1.40
Long30.5631.48+0.9228.70−1.8632.4129.63−2.78
Long output — Qwen3-8B, reasoning
Overallpass@k81.4178.86−2.5677.91−3.5078.8478.18−0.66
avg@k69.4866.65−2.8366.85−2.6367.6367.28−0.35
AIME24pass@886.6786.670.0080.00−6.6783.3383.330.00
avg@877.5075.42−2.0872.08−5.4279.1776.67−2.50
AIME25pass@883.3376.67−6.6680.00−3.3380.0080.000.00
avg@870.8364.17−6.6668.75−2.0865.4267.97+2.55
GPQA-Diamondpass@474.2473.23−1.0173.74−0.5073.2071.21−1.99
avg@460.1060.36+0.2659.72−0.3858.3057.20−1.10
Figure 3. Decode pipelines of dense attention, sparse attention, KV retrieval, and KV prefetching.
Figure 3. Decode pipelines of dense attention, sparse attention, KV retrieval, and KV prefetching.
Figure 4. Top-20 KV-block prediction accuracy across model layers using the previous token vs. a lookahead token.
Figure 4. Top-20 KV-block prediction accuracy across model layers using the previous token vs. a lookahead token.
Table 2. Fetch-cap ablation on Qwen3-8B (AIME24, LRU eviction). The cap bounds the blocks fetched per step by a fetch ratio. Bold marks our default operating point. Dense attention scores 76.04 / 90.00.
Fetch RatioFetchBWTPSAIME24
(GB/step)(GB/s)(tok/s)avg@32pass@32
0.010.305.02,17874.9090.00
0.020.609.82,06675.1090.00
0.051.4923.82,08375.9486.67
0.102.8731.41,42176.7786.67
0.204.3434.01,03577.4093.33
Fetch all5.0533.582476.4686.67
Figure 5. Overview of the OasisKV architecture.
Figure 5. Overview of the OasisKV architecture.
Figure 6. Per-layer agreement between the top-K set predicted by the propagated draft query and the exact set of the true next-token query. The profile uses Qwen3-8B on GSM8K with Tengyunw/qwen3_8b_eagle3 as the EAGLE-3 draft model.
Figure 6. Per-layer agreement between the top-K set predicted by the propagated draft query and the exact set of the true next-token query. The profile uses Qwen3-8B on GSM8K with Tengyunw/qwen3_8b_eagle3 as the EAGLE-3 draft model.

Findings

  • Under a 2,048-token KV budget, accuracy stayed within 0.7 points of full attention.
  • On a reasoning workload, throughput improved 1.69x over dense vLLM at a cost of 0.1 accuracy points.
  • On multi-GPU long-context serving, throughput improved by up to 2.1x.
  • Under prefill-decode disaggregated serving, OasisKV admitted each request with 6.5-9.7x less KV than full transfer, reached about 2x dense throughput, and used 2.2-2.6x less decode-node host memory.
  • On Qwen3-8B, setting the per-step fetch cap to 0.05 kept accuracy within 0.1 point of dense (75.94 vs. 76.04) while achieving 2.5x the throughput of fetching everything (2,083 tok/s).
Figure 7. Look-ahead attention. Left: the CPU full KV cache maps to two GPU caches — compressed keys via per-block min/max pooling and the sparse KV working set via block-wise sparsification. Right: the attention kernel processes the normal and draft queries together over the sparse KV; the draft query then scans the compressed keys to predict the next step’s top-K blocks, which are prefetched from the CPU cache.
Figure 7. Look-ahead attention. Left: the CPU full KV cache maps to two GPU caches — compressed keys via per-block min/max pooling and the sparse KV working set via block-wise sparsification. Right: the attention kernel processes the normal and draft queries together over the sparse KV; the draft query then scans the compressed keys to predict the next step’s top-K blocks, which are prefetched from the CPU cache.
Figure 8. The asynchronous prefetch pipeline across two decoding steps. Red arrows trace one layer’s chain: the draft query at step t drives top-K prediction, KV selection, and KV transfer before that layer’s attention at step t+1.
Figure 8. The asynchronous prefetch pipeline across two decoding steps. Red arrows trace one layer’s chain: the draft query at step t drives top-K prediction, KV selection, and KV transfer before that layer’s attention at step t+1.

Where it can be used

  • Serving infrastructure for chatbots, coding agents, and web-use agents that handle long contexts or long reasoning traces, where reducing GPU memory pressure allows more concurrent requests
  • Throughput improvement for LLM serving systems deployed across multiple GPUs with long-context workloads
  • Reducing network traffic and decode-node host memory in large-scale serving clusters that separate prefill and decode stages
Figure 9. Head-wise mapping between the bounded GPU working set and the CPU full-KV cache. The original page tables retain their logical-to-physical translations. An additional table maps each GPU logical block to a CPU logical block for every KV head.
Figure 9. Head-wise mapping between the bounded GPU working set and the CPU full-KV cache. The original page tables retain their logical-to-physical translations. An additional table maps each GPU logical block to a CPU logical block for every KV head.
Figure 10. Remote partial fetching: a partial transfer at admission (top) and a network fetch at decode (bottom). Numbered steps are described in Sections 4.4.1 and 4.4.2.
Figure 10. Remote partial fetching: a partial transfer at admission (top) and a network fetch at decode (bottom). Numbered steps are described in Sections 4.4.1 and 4.4.2.

Limits and open work

  • Evaluation is limited to specific models (Qwen3-8B, Qwen3-32B, Qwen3-235B) and benchmarks (AIME24/25, GPQA-Diamond, LongBench v2, GSM8K), so generalization to other models or tasks is not yet established.
  • The prototype does not yet support prefix caching; its projected time-to-first-token benefits under prefix caching are estimated analytically rather than measured directly.
  • Reported numbers were measured under specific hardware conditions (H100 GPUs, PCIe bandwidth), and results may differ under other interconnects or memory hierarchies.
  • Prediction misses still occur and require corrective fetches, which partially offset the network traffic savings during decoding.
Figure 11. Synthetic decode sweep over max concurrency at 16K and 32K context, on Qwen3-8B (single H100) and Qwen3-235B (TP8, eight H100s). We compare OasisKV against dense attention on unmodified vLLM (labeled vLLM) and three hierarchical-KV baselines: ShadowKV, InfiniGen, and FreeKV. Rows: decode throughput (TPS), running batch, and per-token latency (TPOT); each request generates 2,048 tokens. Arrows mark the OasisKV/vLLM ratio at max concurrency 256.
Figure 11. Synthetic decode sweep over max concurrency at 16K and 32K context, on Qwen3-8B (single H100) and Qwen3-235B (TP8, eight H100s). We compare OasisKV against dense attention on unmodified vLLM (labeled vLLM) and three hierarchical-KV baselines: ShadowKV, InfiniGen, and FreeKV. Rows: decode throughput (TPS), running batch, and per-token latency (TPOT); each request generates 2,048 tokens. Arrows mark the OasisKV/vLLM ratio at max concurrency 256.
Figure 12. Disaggregated serving over the offered request rate, Qwen3-8B, at 24K (left) and 32K (right) context with 2,048 output tokens. Top: decode throughput. Bottom: average and peak host memory usage in the decode node.
Figure 12. Disaggregated serving over the offered request rate, Qwen3-8B, at 24K (left) and 32K (right) context with 2,048 output tokens. Top: decode throughput. Bottom: average and peak host memory usage in the decode node.

Why it matters

GPU high-bandwidth memory is scarce and expensive, and long-context workloads make it the main bottleneck limiting how many requests a server can handle at once; this work shows a way to keep accuracy nearly intact while serving substantially more requests on the same hardware. It matters especially for multi-GPU deployments and disaggregated serving setups where memory and network costs directly determine how large-scale LLM services can scale.

Terms in this paper

  • KV cache · the stored keys and values from previously processed tokens that a language model must reference to generate each new token
  • HBM · high-bandwidth memory on a GPU that is fast but limited in capacity
  • speculative decoding · a technique that uses a small draft model to guess several future tokens ahead, speeding up generation
  • prefill-decode disaggregation · a serving setup where processing the prompt (prefill) and generating tokens one by one (decode) happen on separate servers
  • capped eviction · a policy that limits how many new KV blocks can be fetched per decoding step to keep data transfer within the available time

Original abstract (English)

Large language model (LLM) inference serving is increasingly constrained by memory rather than compute. As long-context and long-form reasoning workloads become more prevalent, the key-value (KV) cache dominates both memory footprint and memory traffic during LLM token generation, i.e., decode. In particular, HBM capacity has become a scarce and costly resource that heavily limits inference batch size and system throughput. This paper presents OasisKV, a memory-centric LLM inference system design that alleviates HBM capacity pressure by decoupling full KV-cache storage from HBM during LLM decoding. Because decode-time attention is naturally sparse, OasisKV keeps only the KV entries of the most relevant tokens in HBMs for attention computation. We observe that future important tokens can be predicted accurately in advance using lookahead tokens drafted by speculative decoding (SD). OasisKV employs an efficient attention background pipeline to identify important KV blocks. They are then prefetched from higher-capacity memory tiers (e.g., host or remote memory) and staged in HBMs before being used in the next decode step. We implement OasisKV based on vLLM. The lookahead prediction is accurate enough to keep accuracy within 0.7 points of full attention under a 2,048-token KV budget. This lets OasisKV turn sparsity into throughput gain: 1.69times over dense vLLM on the reasoning workload at 0.1 points of accuracy loss, and up to 2.1times on multi-GPU long-context serving. Under prefill--decode disaggregation, OasisKV reaches about 2times dense throughput while admitting each request with 6.5--9.7times less KV and holding 2.2-2.6 less decode-node host memory than full KV transfer.

Authors · Can Xiao

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Can Xiao et al., arXiv:2608.08097, arxiv-nonexclusive