When Do LLMs Actually Help? Evaluating LLMs as Data Quality Annotators

arXiv:2608.181582026-08-20

LLMs aren't a universal fix for data cleaning: their value depends heavily on the task

The study tested GPT-4o-mini on two e-commerce data quality tasks, entity matching and brand mislabeling detection, against rule based baselines. When product names already overlapped textually, the LLM barely beat a simple rule, but when the task needed background knowledge about brands, the LLM clearly won. The model was also remarkably consistent across repeated runs, yet a prompt tweak that looked good on a small sample backfired when applied to the full dataset.

METAL MEDIA explanatory visual

LLMs aren't a universal fix for data cleaning: their value depends heavily on the task

01On the Abt-Buy benchmark (2,194 labeled product pairs), a Jaccard word-overlap rule and zero-shot LLM prompting were compared for deciding whether two listings describe the same product
02On 500 Amazon listings with synthetically swapped manufacturer names, the same comparison was run for detecting mislabeled brands
03For entity matching the rule based baseline (F1=0.950) essentially matched LLM zero-shot prompting (F1=0.948); for brand mislabeling the LLM (F1=0.833) clearly beat the naive rule (F1=0.721)
04A few-shot prompt revision that told the model to prioritize SKU/model codes fixed 28 of 67 errors on a small validation set, but applied to all 2,194 pairs it dropped F1 from 0.948 to 0.914
05Across five repeated runs at temperature 0.7 on a 200-pair sample, the model agreed with itself 99.7% of the time on average, and majority voting across those runs only gained 0.005 F1 at five times the inference cost

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

On the Abt-Buy benchmark (2,194 labeled product pairs), a Jaccard word-overlap rule and zero-shot LLM prompting were compared for deciding whether two listings describe the same product
On 500 Amazon listings with synthetically swapped manufacturer names, the same comparison was run for detecting mislabeled brands
For entity matching the rule based baseline (F1=0.950) essentially matched LLM zero-shot prompting (F1=0.948); for brand mislabeling the LLM (F1=0.833) clearly beat the naive rule (F1=0.721)
A few-shot prompt revision that told the model to prioritize SKU/model codes fixed 28 of 67 errors on a small validation set, but applied to all 2,194 pairs it dropped F1 from 0.948 to 0.914
Across five repeated runs at temperature 0.7 on a 200-pair sample, the model agreed with itself 99.7% of the time on average, and majority voting across those runs only gained 0.005 F1 at five times the inference cost

Fig. 1: Example prompts and responses for the two prompting strategies tested. Panel (a) shows a correctly resolved match under zero-shot prompting; panel (b) shows a true match incorrectly rejected under few-shot SKU-guidance prompting.

TABLE I: Precision, recall, F1, and accuracy for the three entity matching methods evaluated on the Abt-Buy benchmark (2,194 labeled pairs).
Method	Precision	Recall	F1	Accuracy
Rule-based (Jaccard, threshold=0.2)	0.994	0.910	0.950	0.952
LLM zero-shot	0.999	0.902	0.948	0.951
LLM few-shot + SKU guidance	0.994	0.846	0.914	0.920

Fig. 2: Comparison of precision, recall, F1, and accuracy across the three entity matching methods on the Abt-Buy benchmark.

TABLE II: Precision, recall, F1, and accuracy for the two brand mislabeling detection methods evaluated on 500 Amazon product listings.
Method	Precision	Recall	F1	Accuracy
Rule-based (manufacturer-title)	0.564	1.000	0.721	0.622
LLM zero-shot	0.827	0.840	0.833	0.836

Fig. 4: Consistency of LLM predictions across five repeated runs at temperature 0.7 on a 200-pair sample (left), and F1 score for a single deterministic run versus a five-run majority vote on the same sample (right).

Why it matters

For practitioners building data-cleaning pipelines, this gives concrete evidence that LLMs add little value when simple text overlap already solves the problem, and add real value only when the task requires outside knowledge. It also warns that validating a prompt change on a small hand-picked sample can be misleading before scaling it up.

Terms in this paper

zero-shot / few-shot prompting · giving the model a task with no examples (zero-shot) or a few worked examples (few-shot) to guide its answer
F1 score · a metric combining precision and recall into one number, closer to 1 is better
Jaccard similarity · a simple rule that measures how many words two texts share to judge similarity
majority voting · asking the model the same question multiple times and taking the most common answer as the final one
temperature · a setting that controls how random a model's output is; 0 gives the same answer every time, higher values allow more variation

Original abstract (English)

LLMs have been increasingly used to catch data quality issues automatically, but we know very little about how consistent these judgments actually are. This study tests an LLM on two e-commerce data quality tasks, entity matching and brand mislabeling, against rule based baselines and human verified ground truth, under both zero-shot and few-shot prompting. On entity matching while using the Abt Buy benchmark (2,194 labeled pairs), a simple rule based baseline (F1=0.950) performed about as well as LLM zero shot prompting (F1=0.948). Moreover, a few-shot prompt revision that looked effective on a small validation sample reduced full-scale performance to F1=0.914. This showed that small sample prompt evaluation can be misleading. On brand mislabeling detection, using 500 Amazon product listings with synthetically injected labeling errors, the LLM clearly outperformed a naive rule based baseline (F1=0.833 vs 0.721), because it could draw on background knowledge of brand product relationships that a simple rule could not access. Testing consistency across repeated runs (200 pairs, 5 runs at temperature 0.7) showed the model agreeing with itself 99.7% of the time on average, with 99% of pairs giving identical answers across all 5 runs. Using majority voting across these runs only improved F1 by 0.005, at 5 times the inference cost. These results suggest that the value of using an LLM over traditional methods depends heavily on the task. LLMs offer little advantage when strong lexical signals already exist, but a clear advantage when the task requires background knowledge, all while remaining highly consistent across repeated queries.

Authors · Praphulla Lal Shrestha

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

Figures: Praphulla Lal Shrestha et al., arXiv:2608.18158, CC BY 4.0