When Do LLMs Actually Help? Evaluating LLMs as Data Quality Annotators
LLMs aren't a universal fix for data cleaning: their value depends heavily on the task
The study tested GPT-4o-mini on two e-commerce data quality tasks, entity matching and brand mislabeling detection, against rule based baselines. When product names already overlapped textually, the LLM barely beat a simple rule, but when the task needed background knowledge about brands, the LLM clearly won. The model was also remarkably consistent across repeated runs, yet a prompt tweak that looked good on a small sample backfired when applied to the full dataset.
METAL MEDIA explanatory visual
LLMs aren't a universal fix for data cleaning: their value depends heavily on the task
- 01On the Abt-Buy benchmark (2,194 labeled product pairs), a Jaccard word-overlap rule and zero-shot LLM prompting were compared for deciding whether two listings describe the same product
- 02On 500 Amazon listings with synthetically swapped manufacturer names, the same comparison was run for detecting mislabeled brands
- 03For entity matching the rule based baseline (F1=0.950) essentially matched LLM zero-shot prompting (F1=0.948); for brand mislabeling the LLM (F1=0.833) clearly beat the naive rule (F1=0.721)
- 04A few-shot prompt revision that told the model to prioritize SKU/model codes fixed 28 of 67 errors on a small validation set, but applied to all 2,194 pairs it dropped F1 from 0.948 to 0.914
- 05Across five repeated runs at temperature 0.7 on a 200-pair sample, the model agreed with itself 99.7% of the time on average, and majority voting across those runs only gained 0.005 F1 at five times the inference cost
What they did
- On the Abt-Buy benchmark (2,194 labeled product pairs), a Jaccard word-overlap rule and zero-shot LLM prompting were compared for deciding whether two listings describe the same product
- On 500 Amazon listings with synthetically swapped manufacturer names, the same comparison was run for detecting mislabeled brands
- For entity matching the rule based baseline (F1=0.950) essentially matched LLM zero-shot prompting (F1=0.948); for brand mislabeling the LLM (F1=0.833) clearly beat the naive rule (F1=0.721)
- A few-shot prompt revision that told the model to prioritize SKU/model codes fixed 28 of 67 errors on a small validation set, but applied to all 2,194 pairs it dropped F1 from 0.948 to 0.914
- Across five repeated runs at temperature 0.7 on a 200-pair sample, the model agreed with itself 99.7% of the time on average, and majority voting across those runs only gained 0.005 F1 at five times the inference cost
| Method | Precision | Recall | F1 | Accuracy |
|---|---|---|---|---|
| Rule-based (Jaccard, threshold=0.2) | 0.994 | 0.910 | 0.950 | 0.952 |
| LLM zero-shot | 0.999 | 0.902 | 0.948 | 0.951 |
| LLM few-shot + SKU guidance | 0.994 | 0.846 | 0.914 | 0.920 |
| Method | Precision | Recall | F1 | Accuracy |
|---|---|---|---|---|
| Rule-based (manufacturer-title) | 0.564 | 1.000 | 0.721 | 0.622 |
| LLM zero-shot | 0.827 | 0.840 | 0.833 | 0.836 |
Why it matters
For practitioners building data-cleaning pipelines, this gives concrete evidence that LLMs add little value when simple text overlap already solves the problem, and add real value only when the task requires outside knowledge. It also warns that validating a prompt change on a small hand-picked sample can be misleading before scaling it up.
Terms in this paper
- zero-shot / few-shot prompting · giving the model a task with no examples (zero-shot) or a few worked examples (few-shot) to guide its answer
- F1 score · a metric combining precision and recall into one number, closer to 1 is better
- Jaccard similarity · a simple rule that measures how many words two texts share to judge similarity
- majority voting · asking the model the same question multiple times and taking the most common answer as the final one
- temperature · a setting that controls how random a model's output is; 0 gives the same answer every time, higher values allow more variation
Original abstract (English)
LLMs have been increasingly used to catch data quality issues automatically, but we know very little about how consistent these judgments actually are. This study tests an LLM on two e-commerce data quality tasks, entity matching and brand mislabeling, against rule based baselines and human verified ground truth, under both zero-shot and few-shot prompting. On entity matching while using the Abt Buy benchmark (2,194 labeled pairs), a simple rule based baseline (F1=0.950) performed about as well as LLM zero shot prompting (F1=0.948). Moreover, a few-shot prompt revision that looked effective on a small validation sample reduced full-scale performance to F1=0.914. This showed that small sample prompt evaluation can be misleading. On brand mislabeling detection, using 500 Amazon product listings with synthetically injected labeling errors, the LLM clearly outperformed a naive rule based baseline (F1=0.833 vs 0.721), because it could draw on background knowledge of brand product relationships that a simple rule could not access. Testing consistency across repeated runs (200 pairs, 5 runs at temperature 0.7) showed the model agreeing with itself 99.7% of the time on average, with 99% of pairs giving identical answers across all 5 runs. Using majority voting across these runs only improved F1 by 0.005, at 5 times the inference cost. These results suggest that the value of using an LLM over traditional methods depends heavily on the task. LLMs offer little advantage when strong lexical signals already exist, but a clear advantage when the task requires background knowledge, all while remaining highly consistent across repeated queries.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
- Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based one
Latest from METAL MEDIA
Figures: Praphulla Lal Shrestha et al., arXiv:2608.18158, CC BY 4.0