Seeing or Knowing? Visual Context Sensitivity in Multimodal Large Language Models

arXiv:2607.263262026-07-27

Multimodal AIs don't fail to see the image, they fail to control whether they use it

This paper separates why multimodal large language models (MLLMs) fail when an image contradicts their pretrained knowledge into two possible causes. By reconstructing images from only the model's final-layer visual tokens, the authors show the contradictory ('counterfactual') visual information is still there, meaning the problem isn't perception but control over usage. They build a benchmark called WhatIfVis and show through fine-tuning, causal-tracing analysis, and a learned steering vector that this control can be measured and partially recovered.

METAL MEDIA explanatory visual

Perception (H1) vs. Utilization (H2), and the control pipeline

Evidence statusMeasured results reported

1. Reconstruction probeRebuild the image using only the frozen MLLM's final-layer image tokens to check whether counterfactual attributes survive, ruling out H1
2. WhatIfVis evaluationTest vanilla models with paired image/text counterfactual examples under 'use only image' vs. 'ignore image' instructions to expose H2
3. Supervised fine-tuningFine-tune on Spatial-Temporal examples only and check whether controllability transfers to other attribute types
4. Activation patchingSwap internal layer activations in the fine-tuned model to localize the narrow layer band where the vision-versus-prior decision happens
5. Steering vectorFix a learned direction at the located layer to a constant value, controlling image-versus-prior reliance without any instruction in the prompt

An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.

What they did

The authors split the failure of MLLMs to follow instructions about whether to trust an image or their prior knowledge into two hypotheses: perception failure (H1, the image info never arrives) and utilization failure (H2, it arrives but the model can't control whether to use it).
They reconstructed images purely from the frozen (untrained-further) MLLM's final-layer image tokens and had human annotators check whether counterfactual attributes, like an unusual number of legs, survived in the reconstruction.
They built WhatIfVis, a benchmark of 3,049 examples pairing counterfactual images with matched text statements across five attribute types, and tested whether models could follow explicit instructions to 'use only the image' or 'ignore the image and use prior knowledge.'
They applied three interventions across six models: supervised fine-tuning (SFT), activation patching (swapping internal layer computations to trace causal effects), and a learned steering vector that nudges a specific internal direction without any instruction in the prompt.

Figure 2: The counterfactual attribute survives to the final layer and is visible in the reconstruction. Species identity, pose, structure, and dominant colors are preserved, including the counterfactual attribute (e.g. the anomalous limb count), while high-frequency texture softens.

Table 1: Human evaluation of reconstructed images. Across three LLM backbones, the counterfactual attribute (count) is preserved in the vast majority of generated samples. CF denotes counterfactual.
Backbone	CF(%)	Real(%)
Gemma-4-E2B-IT	97.8	2.2
Qwen2.5VL-3B	93.4	6.6
Qwen3.5-4B	97.8	2.2

Figure 3: Evaluation pipeline for controllable multimodal context sensitivity. We probe a model’s sensitivity to visual and textual contexts separately through targeted instruction following. Given a visual (left) or textual (right) context, the model must answer a question by either strictly grounding its reasoning in the explicit context ("Consider only…") or suppressing the context to recall standard world knowledge ("Ignore…").

Table 3: WhatIfVis composition. The benchmark spans five conflict types across two families, incorporating data from VLindBench (16), ViLP (31), Pixel-vs-Priors (10), VLMsAreBiased (45), ROME (57), and a Generated set (49). Perception tasks require reading directly observable attributes; Perception+Reasoning tasks additionally require inference beyond direct visual features.
Family	Task	Description	Sources	# Samples
Perception	Spatial & Temporal	Identify what surrounds, feeds, or characterizes an entity.	VLindBench, ViLP, Generated	VLindBench,	ViLP,	Generated	1,537
VLindBench,
ViLP,
Generated
Color	Report the color of an object shown in a non-canonical hue.	Pixel-vs-Priors, ViLP, VLindBench	Pixel-vs-Priors,	ViLP,	VLindBench	282
Pixel-vs-Priors,
ViLP,
VLindBench
Reasoning	Count	Count parts or instances when the depicted number is anomalous.	VLMsAreBiased, ViLP, VLind-Bench	VLMsAreBiased,	ViLP,	VLind-Bench	390
VLMsAreBiased,
ViLP,
VLind-Bench
Size	Judge relative size when the visual cue inverts the usual ordering.	ROME, Pixel-vs-Priors, VLindBench	ROME,	Pixel-vs-Priors,	VLindBench	773
ROME,
Pixel-vs-Priors,
VLindBench
Weight	Judge relative weight from a balance-scale image with counterfactuals.	Generated	67
Total	3,049

Figure 4: Illustration of the five WhatIfVis tasks across text and vision modalities. Each WhatIfVis example pairs a counterfactual image with a matched text statement asserting the same content. Contextual answers (CTX) are inferred from the visual evidence in the input image, whereas prior answers (PRI) rely on the model’s internal world knowledge. This distinction can reveal the visual context preference of MLLMs.

Table 4: Reconstruction fidelity extends beyond count to the other coarse attributes. Pixel-level agreement between each reconstruction and its counterfactual reference image on the Color, Size, and Weight subsets.
Attribute	Backbone	Counterfactual accuracy%	SSIM↑	PSNR↑	LPIPS↓
Color	Qwen2.5VL-3B	94	0.688	14.39	0.253
Qwen3.5-4B	98	0.718	16.73	0.179
Gemma-4-E2B	98	0.749	18.29	0.146
Size	Qwen2.5VL-3B	92	0.642	13.40	0.308
Qwen3.5-4B	96	0.666	15.14	0.245
Gemma-4-E2B	98	0.720	17.29	0.197
Weight	Qwen2.5VL-3B	92	0.656	16.20	0.246
Qwen3.5-4B	92	0.643	16.04	0.245
Gemma-4-E2B	94	0.687	18.00	0.189

Figure 5: SFT lifts visual controllability across every model and task. Top row: pair-accuracy per task. Bottom row: the same runs as context-accuracy (x) against prior-accuracy (y); points above the diagonal follow the prior over the image. SFT lifts controllability consistently, with the largest gains on Color and Spatial-Temporal and the smallest on Weight and Count. All pair-accuracies are probabilities unless noted.

Table 5: The modality gap is a failure to follow visual evidence, not to override it. Per-intent accuracy (probability scorer, macro-averaged over six models) on the image and matched-text channels. PairAcc credits satisfying both intents on the same example. Here we separate its two halves. Following the evidence (context intent) is far harder from an image than from a matched sentence (+19.0 to +30.8 pp), whereas overriding the evidence (prior intent) is nearly identical across channels (≤4.5 pp, and negative under steering). The gap is thus concentrated in the context intent, the model’s ability to follow visual evidence on demand.
	Context intent (follow evidence)	Prior intent (override evidence)
Condition	Image (%)	Text (%)	Gap (%)	Image (%)	Text (%)	Gap (%)
Base	45.7	73.9	+28.2	68.9	69.5	+0.6
SFT	62.5	93.3	+30.8	88.4	92.9	+4.5
Steering	54.7	73.7	+19.0	77.6	74.7	−2.9

Figure 6: Activation patching localizes the vision-versus-prior trade-off to a narrow layer band. The trade-off initiates within the patched window (marked) and amplifies sharply in downstream layers. Δ(Answer probability)=Sourceprobability−Targetprobability, which is the larger the better patched.

Table 6: Pair-accuracy (probability scorer), averaged over six models The text control is an instruction-following upper bound, since the statement asserts the counterfactual outright. The Random rank-1 projection row applies a randomly-initialized projection at the same image-channel layers with the same multipliers (no DAS; three seeds, per-model std ≤3.5 pp): it falls below the base (−8.8 pp), confirming the steering lift (+23.0 pp) comes from the learned vector, not the freedom of a rank-1 edit.
Condition	Image (%)	Δ vs. base (pp)	Text (%)	Text−Image (pp)
Baseline	14.7	—	43.4	28.6
Random rank-1 projection (image)	5.9	−8.8	—	—
SFT	52.7	+38.0	86.3	33.6
Steering	37.7	+23.0	53.3	15.6

Findings

Across three different model families (Qwen2.5VL-3B, Qwen3.5-4B, Gemma-4-E2B-IT), reconstructions correctly recovered the counterfactual leg count in 93.4%, 97.8%, and 97.8% of samples respectively.
Vanilla (not fine-tuned on the task) models had very low pair-accuracy, satisfying both the 'use image' and 'ignore image' instructions on the same example only 14.7% of the time on average across six models.
Fine-tuning on just the Spatial-Temporal attribute lifted average pair-accuracy from 14.7% to 52.7% across six models, with Color and Spatial-Temporal approaching 80%, and the improvement transferred to other held-out attribute types.
Activation patching localized the vision-versus-prior trade-off to a narrow band of layers in all six models, though the exact depth differed by model architecture.
Applying a learned steering vector with no instruction at all raised average pair-accuracy from 14.7% to 37.7%, while a randomly initialized vector applied the same way dropped it to 5.9%, below baseline, confirming the effect depends on the specific learned direction.

Where it can be used

Services relying on multimodal models to prioritize either visual evidence or background knowledge on demand, such as anomaly detection or fact-checking chatbots, could use similar diagnostic methods to check controllability.
Prompt design or output verification for tasks involving inference-heavy attributes (like counting or weight comparison) should account for the finding that models tend to default to prior knowledge over visual evidence on these attributes.
The finding that fine-tuning on a small dataset (1,136 samples) for one attribute partially transfers control to other attributes suggests a data-efficient strategy for improving visual-instruction adherence.

Limits and open work

The reconstruction experiments and benchmark are limited to coarse-grained attributes (color, count, size, weight, spatial-temporal), not fine details like texture or small objects.
Attributes requiring inference beyond direct visual reading, like Count and Weight, showed much smaller improvements from fine-tuning or steering.
The steering vector's location was identified using fine-tuned models and then applied back to vanilla models; this does not mean the vanilla models were inherently steerable on their own.
Text-based control was consistently more reliable than image-based control, and this gap widened with model scale, but the underlying reason is left as future work.
The study covers only three model families and six models total, so it's unverified whether these findings generalize to other architectures or larger models.

Figure 7: An instruction-free steering knob recovers controllability over the vanilla model, and the visual channel stays harder to control than text. Per-model macro pair-accuracy (probability scorer); left: image, right: text. Steering lifts every model; text is uniformly higher (Appendix E).

Why it matters

It was widely assumed MLLMs fail on such tasks because their vision encoders lose detail, but this work shows the visual information is often already present internally and the real bottleneck is the model's inability to reliably decide when to trust it versus its prior knowledge. This reframes where future fixes should focus: not just on better vision encoders, but on the internal decision mechanism governing image-versus-prior reliance.

Figure 8: Metaquery Framework. Without learnable queries, we adapt the Metaquery to only use the image tokens of the final layers of a MLLM.

Terms in this paper

MLLM (Multimodal Large Language Model) · An AI model that takes both images and text as input and generates answers
Activation patching · A technique that swaps the internal computation at a specific model layer with values from a different input, to test how much that layer influences the final output
Steering vector · A learned direction in the model's internal representation that, when forced to a fixed value, changes the model's behavior (e.g., toward image or toward prior knowledge) without needing any instruction in the prompt
PairAcc (pair-accuracy) · A strict metric that only counts an example correct if the model satisfies both the 'follow the image' and 'follow the prior' instructions on that same example
Counterfactual image · An image showing something that contradicts common-sense expectations, such as an animal with six legs

Figure 9: The SFT lift reproduces under exact-match at lower absolute values. Per-task pair-accuracy (exact-match scorer, image channel). Orange: baseline (with intent instructions); blue: SFT (LoRA, trained on Spatial-Temporal only; all other tasks are out-of-distribution). The qualitative pattern, with large gains on the perceivable attributes and marginal gains on the reasoning-heavy ones, matches the probability scorer (Figure 5).

Figures we cannot republish

Figure 1: One MLLM, two opposite failures. A single model lacks a stable setpoint on the visual context-sensitivity axis between its parametric prior and the image, giving rise to two opposite failure modes: (Left) Prior-dominant failures, where strong parametric priors overwhelm anomalous visual input; and (Right) Context-dominant failures, where the model over-indexes on visual features even when instructed to ignore them..

See the figures in the original paper →

Original abstract (English)

Multimodal Large Language Models (MLLMs) achieve strong performance by integrating visual inputs with the rich priors of pretrained language models. However, they often fail on vision-centric tasks, especially when visual evidence conflicts with pretrained knowledge. We explore these failures separately using two diagnostic paradigms: (1) probing whether visual information is available, via image reconstruction, and (2) measuring multimodal context sensitivity, the extent to which the model foll

Authors · Jiaang Li

Read on arXiv

Latest papers

All papers →

Latest from METAL MEDIA

NCT 127 drop BLINGY, their first full album in two years

Figures: Jiaang Li et al., arXiv:2607.26326, CC BY 4.0