You Are What You Prompt: Prompt Quality, Domain Shift, and Uncertainty in Agrifood Vision-Language Models
Why asking an AI about food photos and plant disease photos the same way makes it fail on the disease ones
Vision-language models like CLIP and SigLIP classify images by matching them to text prompts, and combining many prompts (prompt ensembling) is known to help, but this paper shows it helps very differently depending on the domain: strongly on food images, barely on plant disease images. The authors study Zero-shot Prompt Ensembling (ZPE), a label-free method that scores how useful each prompt is, and introduce PID, a new method that uses disagreement between prompts to flag when the model is likely wrong. On food data the model's own confidence was already reliable, but on the unfamiliar bean-disease dataset PID caught errors far better than the model's raw confidence.
METAL MEDIA explanatory visual
Why asking an AI about food photos and plant disease photos the same way makes it fail on the disease ones
- 01The authors tested six models across two families (CLIP and SigLIP) on two food datasets (Food-101, Food-11) and two agriculture datasets (Agricultural crops, Beans leaf disease).
- 02ZPE, which weights prompts by a label-free discriminative signal, gave only marginal gains on food images (close to what the model was trained on) but substantially improved accuracy on the out-of-distribution Beans dataset.
- 03A word-level analysis found that concrete, viewpoint-related words like 'aerial' and 'view' were associated with high-quality prompts, while abstract words like 'type' or 'demonstration' were linked to poor prompts.
- 04Prompt quality rankings for food and for agriculture were strongly correlated (ρ=0.765), suggesting that a prompt's usefulness partly transfers across these two related domains.
- 05The new PID method, which measures disagreement across prompts, achieved AUROC 0.646 versus 0.275 for the model's raw confidence (max-conf) on CLIP-B/16 with Beans data — a +0.37 gap, the largest improvement observed in the whole study.
What they did
- The authors tested six models across two families (CLIP and SigLIP) on two food datasets (Food-101, Food-11) and two agriculture datasets (Agricultural crops, Beans leaf disease).
- ZPE, which weights prompts by a label-free discriminative signal, gave only marginal gains on food images (close to what the model was trained on) but substantially improved accuracy on the out-of-distribution Beans dataset.
- A word-level analysis found that concrete, viewpoint-related words like 'aerial' and 'view' were associated with high-quality prompts, while abstract words like 'type' or 'demonstration' were linked to poor prompts.
- Prompt quality rankings for food and for agriculture were strongly correlated (ρ=0.765), suggesting that a prompt's usefulness partly transfers across these two related domains.
- The new PID method, which measures disagreement across prompts, achieved AUROC 0.646 versus 0.275 for the model's raw confidence (max-conf) on CLIP-B/16 with Beans data — a +0.37 gap, the largest improvement observed in the whole study.
| Model | F101 | F11 | Agri. | Beans | Avg |
|---|---|---|---|---|---|
| CLIP-B/32 | 0.830 | 0.809 | 0.616 | 0.287 | 0.636 |
| CLIP-B/16 | 0.881 | 0.810 | 0.709 | 0.307 | 0.677 |
| CLIP-L/14 | 0.932 | 0.848 | 0.780 | 0.398 | 0.740 |
| SigLIP-B | 0.912 | 0.844 | 0.869 | 0.404 | 0.757 |
| SigLIP-L | 0.943 | 0.845 | 0.910 | 0.635 | 0.833 |
| SigLIP-400M | 0.957 | 0.842 | 0.941 | 0.635 | 0.844 |
Why it matters
For anyone deploying general-purpose vision-language AI in specialized areas like agriculture or food safety, this work offers a practical, label-free way to tell when the model's outputs can be trusted and when they should be double-checked. It shows that a model's own confidence score, which works well in familiar domains, becomes misleading exactly where mistakes matter most — under domain shift.
Terms in this paper
- zero-shot classification · classifying images using text descriptions alone, without training on labeled examples for those exact classes
- prompt ensembling · combining predictions from many different text prompt wordings for the same class
- ZPE (Zero-shot Prompt Ensembling) · an existing method that scores and weights prompts by their discriminative power, without needing ground-truth labels
- PID (Prompt-based Inconsistency Detection) · the paper's new method that measures how much prompts disagree with each other to predict likely model errors
- out-of-distribution (OOD) · data that looks very different from what the model saw during training
- AUROC · a score measuring how well a signal distinguishes correct from incorrect predictions, higher is better
Figures we cannot republish
- Figure 1: Zero-shot performance for SigLIP SO400M and CLIP ViT-L/14 (all prompt pools).
- Figure 2: Word discriminability of top vs. bottom 25% ZPE-ranked prompts per dataset (SigLIP-SO400M, Pool-247).
- Figure 3: Cross-domain ZPE-norm prompt scores: Food vs. Agriculture (SigLIP-SO400M, Pool-247). ρ=0.765 indicates strong cross-domain prompt quality transferability.
- Figure 4: PID-based uncertainty evaluation across all prompt pools.
- Figure 5: PID AUROC as a function of τa across all datasets and prompt pools. Peaked curves (Food) indicate quality-gating is beneficial; monotonically increasing curves (Beans with CLIP) indicate full domain shift.
Original abstract (English)
Vision-language models enable zero-shot classification through natural language prompts, but performance is sensitive to prompt formulation, especially in specialized domains. Zero-shot Prompt Ensembling (ZPE) addresses this by weighting prompts by discriminative signal, yet its behavior under domain shift remains unexplored. We evaluate ZPE in the agrifood domain using CLIP and SigLIP across four datasets and four prompt pools, spanning in-distribution (ID) food and out-of-distribution agricultural benchmarks. ZPE provides limited benefit under ID conditions but substantially improves performance and calibration under domain shift, where domain-specific pools of 51-52 prompts consistently outperform generic pools of 247-426. Lexical analysis shows that ZPE acts as an unsupervised domain-alignment detector without label access. We further introduce PID (Prompt-based Inconsistency Detection), which repurposes prompt disagreement as epistemic uncertainty, improving failure detection under severe domain shift where standard confidence measures collapse.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctly
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears