SESSE: Sketch, Expand, Sort, Summarize, Evaluate -- LLM-as-Judge Evaluation via Structured Decomposition
Making AI judges show their work when picking the better of two answers
Today's common way to grade AI outputs is asking another AI (an LLM judge) to pick A or B, with no way to see why it picked that way or whether disagreement means the judge was wrong or the case was genuinely ambiguous. SESSE mines specific evaluation sub-questions automatically from the judge's own past mistakes, groups them into clusters, and has the judge vote A/B/Not-Applicable on each relevant sub-question to produce both a final decision and a per-criterion audit trail, all without any training. On the RewardBench benchmark, SESSE matched the accuracy of standard chain-of-thought judging and came close to a fine-tuned specialist model, despite requiring no training at all.
METAL MEDIA explanatory visual
Making AI judges show their work when picking the better of two answers
- 01Standard LLM-judge evaluation collapses everything into one A/B choice, giving no way to isolate which quality dimension drove the decision or tell a judge's mistake apart from a genuinely ambiguous case.
- 02SESSE collects cases where the judge's initial holistic verdict disagreed with the ground-truth label on a development set, and automatically generates evaluation sub-questions from those error cases, clustering them into 25 groups to build a reusable question bank.
- 03At inference time, only relevant clusters are selected for each example, the judge votes A/B/NA on each, and the majority vote across all selected clusters becomes the final preference; equal votes are declared a tie.
- 04On the 1,000-example RewardBench validation set, SESSE with Gemini 2.5 Flash reached 93.3% accuracy, statistically indistinguishable from the plain chain-of-thought baseline, and comparable to the fine-tuned specialist RISE-Judge-32B (92.7%).
- 05Weaker judge models (Qwen2-VL-7B) showed a larger, statistically significant accuracy gap versus the baseline, and a high tie rate (21.3% vs 2.6-3.5% for Gemini models) served as a warning sign of lower judge reliability without needing ground-truth labels.
What they did
- Standard LLM-judge evaluation collapses everything into one A/B choice, giving no way to isolate which quality dimension drove the decision or tell a judge's mistake apart from a genuinely ambiguous case.
- SESSE collects cases where the judge's initial holistic verdict disagreed with the ground-truth label on a development set, and automatically generates evaluation sub-questions from those error cases, clustering them into 25 groups to build a reusable question bank.
- At inference time, only relevant clusters are selected for each example, the judge votes A/B/NA on each, and the majority vote across all selected clusters becomes the final preference; equal votes are declared a tie.
- On the 1,000-example RewardBench validation set, SESSE with Gemini 2.5 Flash reached 93.3% accuracy, statistically indistinguishable from the plain chain-of-thought baseline, and comparable to the fine-tuned specialist RISE-Judge-32B (92.7%).
- Weaker judge models (Qwen2-VL-7B) showed a larger, statistically significant accuracy gap versus the baseline, and a high tie rate (21.3% vs 2.6-3.5% for Gemini models) served as a warning sign of lower judge reliability without needing ground-truth labels.
| System | Holistic | SESSE | Tie% | |
|---|---|---|---|---|
| non-tie | tie | non-tie | ||
| Qwen2-VL-7B | 0.698 | 0.667 | 0.663‡ | 21.3% |
| Flash Lite | 0.880 | 0.679 | 0.820‡ | 2.6% |
| Gemini 2.5 Flash | 0.945 | 0.695 | 0.933 | 3.5% |
| RISE-Judge (32B)† 18 | 0.927 | — |
| n | Qwen2-VL-7B | Gemini 2.5 Flash | Flash Lite |
|---|---|---|---|
| 1 | 0.626 | 0.923 | 0.790 |
| 2 | 0.648 | 0.933 | 0.804 |
| 3 | 0.663 | 0.927 | 0.801 |
| 5 | 0.644 | 0.932 | 0.820 |
| 7 | 0.645 | 0.929 | 0.817 |
| 10 | 0.643 | 0.930 | 0.819 |
| Config | keff | SESSE acc (n=10) | SESSE (n∗) |
|---|---|---|---|
| Qwen self-generated | 8 | 0.640 | 0.663 (n∗=3) |
| Qwen+Gemini bank | 14 | 0.637 | 0.663 (n∗=2) |
| Config | Judge | Bank | Degradation (% val) | C1 (Ambiguous) | C2 (Decomposition) | C3 (Semantic equiv.) |
|---|---|---|---|---|---|---|
| Flash Lite | Flash Lite | self-generated | 105 (10.5%) | 49% | 26% | 26% |
| Gemini 2.5 Flash | Gemini 2.5 Flash | self-generated | 48 (4.8%) | 29% | 23% | 48% |
Why it matters
Being able to see exactly which criteria drove an AI judge's verdict lets practitioners catch mislabeled benchmark data and judge biases instead of blindly trusting a single opaque score. Because it requires no fine-tuning, this diagnostic capability can be added on top of existing LLM judges at essentially no extra training cost.
Terms in this paper
- LLM-as-judge · using a large language model instead of a human to evaluate the quality of another AI's output
- chain-of-thought (CoT) · a prompting technique where the model writes out step-by-step reasoning before giving a final answer
- RewardBench · a benchmark dataset of paired AI responses used to test preference-judging accuracy
- NA (Not Applicable) vote · an option the judge selects when a given evaluation criterion doesn't apply to the example
- McNemar's test · a statistical test comparing two methods only on the cases where their answers disagree
Figures we cannot republish
- Figure 1: SESSE replaces holistic judgment with structured decomposition — criteria emerge automatically from the judge’s own error cases, producing per-criterion vote evidence.
- Figure 2: SESSE pipeline: offline bank construction (Stages 0–4, once) and online per-example inference (Stage 5).
Original abstract (English)
LLM-as-judge evaluation reduces response quality assessment to a single holistic A/B preference choice, providing no mechanism to isolate which quality dimensions drove the preference or distinguish model errors from genuine label ambiguity. We propose SESSE (Sketch, Expand, Sort, Summarize, Evaluate), a training-free framework that decomposes holistic judgment into structured sub-questions mined directly from the judge's own error cases; requiring no oracle responses, task-specific rubrics, or fine-tuning. On RewardBench (n=1,000), SESSE achieves near-parity with the chain-of-thought baseline and is competitive with RISE-Judge-32B (92.7%), a fine-tuned specialist, while remaining fully training-free. Per-criterion vote evidence provides an interpretable audit trail for diagnosing label ambiguity and judge failure modes unavailable from a single holistic output token.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
- Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based one