Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
Fractional Decay KV-Cache: Ownership-Aware Memory Management for Improved Inference Relevancy in Dialog Systems
arXiv:2608.180982026-08-20
Keeping a chatbot's memory from clinging to yesterday's topic
Dialog AI systems store past conversation turns in a temporary memory called a KV-cache, but existing methods keep clinging to old important-seeming entries even after the topic has moved on. This paper proposes FD-KVC, which scores each cached entry with both a slow-fading cumulative importance score and a fast-decaying recency score, letting genuinely useful old context survive while stale content fades quickly. Across five dialog benchmarks, it adapts to topic shifts 3.6x faster than the leading method H2O and scores 6.7% higher on a composite relevance metric.
METAL MEDIA explanatory visual
Keeping a chatbot's memory from clinging to yesterday's topic
01As dialogs grow longer, the cache fills up and older but no-longer-relevant entries dilute attention; the existing state-of-the-art method H2O suffers from a 'stale cache' problem because its cumulative attention score only ever increases and can't shrink when topics change
02FD-KVC gives each cached entry two scores: a cumulative attention score (that persists like H2O's) and a recency score that decays every turn but gets reinforced when the entry matches the current query, then blends both into a single 'ownership' score used for deciding what to evict
03An adaptive learning rate tied to a convergence-tracking loss keeps the scoring stable, stabilizing within 2-3 turns, and the whole method runs on plain CPU at 0.105 milliseconds per turn with no GPU needed
04Tested on 600 dialogs each across five scenarios, FD-KVC beat H2O by +6.7% on composite late-turn alignment, with gains of +127% on abrupt topic shifts, +87% on gradual topic drift, and +30% on mixed-topic dialogs, plus the highest topic diversity retained (80.6%)
05It underperforms H2O by 39% when a topic returns after being dropped, since decayed information can't be recovered, and all experiments used synthetic 64-dimensional embeddings rather than a real language model
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
As dialogs grow longer, the cache fills up and older but no-longer-relevant entries dilute attention; the existing state-of-the-art method H2O suffers from a 'stale cache' problem because its cumulative attention score only ever increases and can't shrink when topics change
FD-KVC gives each cached entry two scores: a cumulative attention score (that persists like H2O's) and a recency score that decays every turn but gets reinforced when the entry matches the current query, then blends both into a single 'ownership' score used for deciding what to evict
An adaptive learning rate tied to a convergence-tracking loss keeps the scoring stable, stabilizing within 2-3 turns, and the whole method runs on plain CPU at 0.105 milliseconds per turn with no GPU needed
Tested on 600 dialogs each across five scenarios, FD-KVC beat H2O by +6.7% on composite late-turn alignment, with gains of +127% on abrupt topic shifts, +87% on gradual topic drift, and +30% on mixed-topic dialogs, plus the highest topic diversity retained (80.6%)
It underperforms H2O by 39% when a topic returns after being dropped, since decayed information can't be recovered, and all experiments used synthetic 64-dimensional embeddings rather than a real language model
Table 1: Late-turn alignment across five benchmarks (600 dialogs each, budget=56). Bold indicates best per column. Comp = composite mean.
Method
Shift
Return
Mixed
Cplx
Grad
Comp
FIFO
.0200
.0283
.0206
.0301
.0200
.0238
Sink+W
.0189
.0293
.0206
.0302
.0190
.0236
H2O
.0084
.0320
.0151
.0285
.0079
.0184
FD-KVC
.0190
.0194
.0197
.0253
.0148
.0196
Table 2: Adaptation speed (turns to 80% new-topic retention after A→B shift). H2O never adapts within 16 turns.
Method
Mean
Median
p90
FIFO
2.0
2
2
Sink+W
2.0
2
2
H2O
16.0
16
16
FD-KVC
4.5
4
5
Table 3: Late-turn topic diversity (%). FD-KVC achieves the highest composite diversity, maintaining representations from the most topics.
Method
Shift
Return
Mixed
Cplx
Comp
FIFO
50.0
66.7
44.4
26.7
47.6
Sink+W
100.0
66.7
66.7
33.3
73.3
H2O
100.0
50.0
66.7
33.3
70.0
FD-KVC
72.1
100.0
73.1
58.0
80.6
Table 4: Composite late-turn topic retention and average latency.
Method
LateRet (%)
Latency (ms)
FIFO
91.4
0.027
Sink+W
87.6
0.028
H2O
63.3
0.055
FD-KVC
70.1
0.105
Table 5: Ablation on cumulative weight wc (Mixed, 200 dialogs). Peak alignment at wc=0.3.
wc
LateAlign
LateRet (%)
0.1
+0.0223
77.6
0.2
+0.0455
75.4
0.3
+0.0483
72.8
0.5
+0.0238
60.2
0.7
+0.0081
29.9
0.9
+0.0063
00.1
Table 6: Ablation on decay rate γ (Mixed, 200 dialogs). Faster decay (γ≈0.80) yields best alignment.
γ
LateAlign
LateRet (%)
0.70
+0.0233
85.7
0.75
+0.0246
83.8
0.80
+0.0603
84.7
0.85
+0.0097
85.6
0.88
+0.0148
64.1
0.92
−0.0062
27.3
Table 7: FD-KVC hyperparameter configuration.
Param
Default
Range
Description
γ
0.88
[0.70, 0.95]
Recency decay rate
α0
0.30
[0.10, 0.50]
Initial learning rate
τ
0.02
[0.01, 0.10]
Eviction threshold
β
0.25
[0.10, 1.00]
Modulation exponent
μ
0.40
[0.10, 1.00]
Convergence factor
wc
0.45
[0.10, 0.90]
Cumulative weight
wρ
0.55
[0.10, 0.90]
Recency weight
Why it matters
This shows a way to stop dialog AI from getting stuck on outdated conversation topics without retraining the underlying model, which matters for anyone building long-running chat assistants where topics naturally shift. Because it runs on CPU with negligible overhead, it's a low-cost technique to consider, though it still needs validation on real language models and real dialog data before production use.
Terms in this paper
KV-cache · a temporary memory that stores earlier conversation representations so a transformer model doesn't have to recompute them
H2O · an existing cache method that keeps 'heavy-hitter' tokens with the highest cumulative attention scores
cumulative attention score · a running total of how much attention an entry has ever received, which tends to only go up
recency score · a score reflecting how recently and currently relevant an entry is, which fades over time unless reinforced
FIFO · a simple cache policy that evicts the oldest entries first
Figures we cannot republish
Figure 1: Per-turn topic alignment for three benchmarks. After topic shifts, H2O alignment drops (stale cache), while FD-KVC adapts.
Figure 2: Left: Ownership loss convergence. Right: Adaptive learning rate. Loss stabilizes within 2–3 turns.
Figure 3: Topic retention under topic shift. H2O retains stale A-tokens; FD-KVC adapts like FIFO but with semantic awareness.
Figure 4: Ablation curves for wc (left) and γ (right). Red: alignment; blue dashed: retention.
Key-value (KV) caching is essential for efficient autoregressive inference in transformer based dialog systems, yet existing strategies treat all cached entries uniformly or apply coarse eviction heuristics that fail to adapt as dialog topics evolve. We propose Fractional Decay KV-Cache (FD-KVC), a novel algorithm that maintains a dual-channel scoring mechanism for each cached KV pair: a cumulative attention channel that tracks aggregate importance (akin to H2O), and a recency-weighted relevance channel governed by temporal decay and reinforcement-inspired updates. The combination enables FD-KVC to both preserve historically important tokens and rapidly adapt when dialog topics shift. An adaptive learning rate driven by an ownership loss function ensures convergence without oscillation. FD-KVC operates entirely on CPU with negligible overhead. Across five diverse multi-turn dialog scenarios with 600 dialogs each, FD-KVC outperforms H2O, the state-of-the-art heavy-hitter baseline, by +6.7% on composite late-turn alignment, with improvements of +127% on topic-shift, +87% on gradual evolution, and +30% on mixed-topic dialogs. FD-KVC adapts to new topics 3.6X faster than H2O and achieves the highest topic diversity (80.6%) across all methods. Ablation studies confirm the contribution of each component.
