Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox

METAL MEDIA

Pairwise Ranking Outperforms Single-Action RL for Offline Explanation Selection: A Practical Lesson

arXiv:2608.185312026-08-20

You don't need to generate explanation text on every request - pre-make it and just pick one

Recommender systems that show LLM-written reasons for each suggestion pay a real cost: every request triggers an expensive, slow LLM call. This paper pre-generates a pool of candidate explanations offline and uses a tiny CPU-only selector to pick one at request time. Surprisingly, a simple pairwise ranking method beat several reinforcement-learning approaches at choosing the best candidate.

METAL MEDIA explanatory visual

You don't need to generate explanation text on every request - pre-make it and just pick one

  1. 01Using Google Local reviews and MovieLens-1M data, the authors pre-generated explanation candidates with six prompt styles across two low-cost LLMs (Amazon Nova Lite and Claude Haiku)
  2. 02They compared nine ways to pick the best candidate: a pairwise learning-to-rank method (LambdaRank), reinforcement learning methods (PPO, GRPO, DPO), knowledge-graph path-based selection, and teacher-student distillation
  3. 03LambdaRank scored 0.500 BERTScore-F1 on Google Local, beating published baselines G-Refer (0.459) and XRec (0.431), with results stable across five repeated runs (variance under 0.003)
  4. 04The RL methods (PPO, GRPO, DPO) each only learn from the reward of one sampled candidate per round, while LambdaRank uses the labeled score of all 40 candidates at once - this difference explains why LambdaRank won
  5. 05Knowledge-graph path-based methods hit near-perfect diversity scores (USR close to 1.000), making them better suited when varied output matters more than matching a reference text exactly
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

  1. Using Google Local reviews and MovieLens-1M data, the authors pre-generated explanation candidates with six prompt styles across two low-cost LLMs (Amazon Nova Lite and Claude Haiku)
  2. They compared nine ways to pick the best candidate: a pairwise learning-to-rank method (LambdaRank), reinforcement learning methods (PPO, GRPO, DPO), knowledge-graph path-based selection, and teacher-student distillation
  3. LambdaRank scored 0.500 BERTScore-F1 on Google Local, beating published baselines G-Refer (0.459) and XRec (0.431), with results stable across five repeated runs (variance under 0.003)
  4. The RL methods (PPO, GRPO, DPO) each only learn from the reward of one sampled candidate per round, while LambdaRank uses the labeled score of all 40 candidates at once - this difference explains why LambdaRank won
  5. Knowledge-graph path-based methods hit near-perfect diversity scores (USR close to 1.000), making them better suited when varied output matters more than matching a reference text exactly
  6. The whole selector runs on CPU under 100 milliseconds per request, with total build cost around $15
Figure 2. Adaptive entropy schedule during PPO training on Google Local (seed 42). Top: entropy coefficient β. Bottom: action-distribution entropy H. The scheduler bumps β up when reward improvement stalls and down when it accelerates; the policy converges to a near-deterministic selector by episode 500.
Figure 2. Adaptive entropy schedule during PPO training on Google Local (seed 42). Top: entropy coefficient β. Bottom: action-distribution entropy H. The scheduler bumps β up when reward improvement stalls and down when it accelerates; the policy converges to a near-deterministic selector by episode 500.
Table 1. Main results on Google Local and MovieLens-1M. Higher is better for BERTScore-F1; less negative is better for BARTScore; USR (Unique-Sentence Ratio) lies in [0,1], higher is more diverse. Best score per column is in bold. Note on “±”: KG-path stds are per-sample on a single trained policy; offline-pool stds are across 5 training seeds. Cached-prediction baselines (XRec, G-Refer) and our pool-only heuristic have no seed variance. “–” marks metrics not scored for the row. Numbers for XRec and G-Refer are taken verbatim from their published Table 2 / Table 1 respectively. KG-path family rows are reproduced from concurrent work by a co-author; offline-pool family rows are computed in this work on the n=2,958 review-covered Google Local subset and the n=300 MovieLens-1M test split.
Google LocalMovieLens-1M
MethodBERTScore (F1) ↑BART Score ↑USR ↑BERTScore (F1) ↑BART Score ↑USR ↑
Published baselines (numbers taken from each paper)
XRec (Ma et al. 2024)0.4311−4.16470.9993–––
G-Refer 8B (Li et al. 2025)0.4592−3.32351.0000–––
KG-path family (concurrent work)
Temperature-biased walks0.3258 ± 0.075−3.5761.00000.2690 ± 0.068−3.6291.000
Edge-disjoint enumeration0.3265 ± 0.074−3.5771.00000.2703 ± 0.069−3.6131.000
MMR paths + dual-style0.3252 ± 0.075−3.5771.00000.2621 ± 0.070−3.6240.997
Offline-pool family (this work)
Pool-only heuristic0.4444––0.2634––
PPO (5 seeds)0.4581 ± 0.001−3.3540.9760.2816 ± 0.003−3.5660.999
GRPO (5 seeds)0.4703 ± 0.001−3.3540.9510.2830 ± 0.002−3.5330.999
DPO (5 seeds)0.4749 ± 0.001−3.3740.9090.2936 ± 0.002−3.5300.999
Distillation A+B (5 seeds)0.4767 ± 0.001−3.3560.9250.2831 ± 0.003−3.5440.999
Distillation A (5 seeds)0.4817 ± 0.000−3.3750.8650.2887 ± 0.001−3.5481.000
LambdaRank0.5003−3.3270.8080.3291−3.4490.987
Figure 3. Generator-pool robustness on Google Local: Haiku 3 (grey) vs. Haiku 4.5 (blue), same selector. Method ordering is preserved; LambdaRank still leads on both pools. Dotted line: G-Refer 8B baseline.
Figure 3. Generator-pool robustness on Google Local: Haiku 3 (grey) vs. Haiku 4.5 (blue), same selector. Method ordering is preserved; LambdaRank still leads on both pools. Dotted line: G-Refer 8B baseline.
Table 2. Generator upgrade, Google Local (n=2,958). “Haiku 3” columns report the absolute score on the Claude Haiku 3 candidate pool (the pool used in Table 1); Δ columns report the change when the pool is regenerated with Claude Haiku 4.5 holding the selector fixed (negative = upgrade hurts). BERTScore-F1 drops 0.001–0.006 on every method; BARTScore shifts within ±0.007 (effectively flat); USR improves by +0.018–+0.083 because Haiku 4.5’s outputs are less templated than Haiku 3’s.
BERTScore (F1) ↑BART Score ↑USR ↑
MethodHaiku 3ΔHaiku 3ΔHaiku 3Δ
PPO0.4581−0.001−3.354−0.0020.976+0.018
GRPO0.4703−0.003−3.354+0.0050.951+0.036
DPO0.4749−0.006−3.374+0.0060.909+0.066
Distill A+B0.4767−0.003−3.356+0.0070.925+0.050
Distill A-only0.4817−0.004−3.375+0.0030.865+0.083
LambdaRank0.5003−0.002−3.327−0.0030.808+0.074

Why it matters

In production, generating explanation text with an LLM on every single request drives up latency and cost as traffic grows; this work shows you can separate generation from selection to get GPU-free, fast, cheap responses without sacrificing quality. It also offers a practical lesson: try simple ranking methods as a baseline before reaching for reinforcement learning.

Terms in this paper

  • LLM · a large language model that generates text
  • BERTScore-F1 · a score measuring how semantically similar generated text is to a reference text
  • LambdaRank · a pairwise learning-to-rank method that learns relative ordering among candidates
  • PPO/GRPO/DPO · reinforcement learning or preference-learning methods that update based on one sampled outcome at a time
  • USR (Unique-Sentence Ratio) · a metric measuring how varied and non-repetitive generated sentences are

Original abstract (English)

Industrial explainable-recommendation systems built on LLMs incur a substantial serving cost: each request triggers an LLM generation, with latency in the hundreds of milliseconds and cost that scales linearly with traffic. We separate generation from selection: explanations are produced ahead of time as a frozen candidate pool (six prompt styles, two commodity LLMs), and a small CPU-resident selector picks one at request time. The stack needs no GPU and returns in under 100 ms. Our primary benchmark is a 2,958-pair XRec Google Local subset, evaluating six offline-pool selectors (LambdaRank, PPO, GRPO, DPO, teacher-student distillation) and three KG-path selectors (random walks, edge-disjoint enumeration, MMR-reranked paths). A 300-pair MovieLens-1M split with Claude-Sonnet-4.5 references serves as an internal cross-dataset check, since no public benchmark exists for this setting. All variants use the same BERTScore-F1 protocol as XRec and G-Refer, averaged across five seeds. LambdaRank reaches F1 = 0.500 on Google Local, exceeding both G-Refer and XRec, and F1 = 0.329 on the MovieLens-1M check. With seed variance below 0.003 F1, the ordering is reliable: pairwise learning-to-rank outperforms single-action RL (PPO, GRPO, DPO), which use only one labelled candidate per rollout, leaving K-1 labels unused. The KG-path family targets a different objective: all three variants reach USR = 1.000 on Google Local and 0.997-1.000 on MovieLens-1M, since per-request path grounding yields a unique output per query, avoiding template-collapse failures affecting cached-LLM outputs. A generator-pool study comparing Claude 3 Haiku and Claude Haiku 4.5 shows small F1 shifts (0.001-0.006) while preserving selector ranking: selector and generator can be evaluated independently, though absolute F1 depends on the generator. End-to-end build cost is near $15 on commodity hardware.

Authors · Tanay Chowdhury, Saeideh Shahrokh Esfahani

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Tanay Chowdhury et al., arXiv:2608.18531, CC BY 4.0