OmniScope: Modality-Decoupled Token Compression for Omnimodal Large Language Models
In AI models that watch video and listen to audio at once, letting one sense decide for the other can throw away the answer
Omnimodal large language models that jointly process audio and video need to cut down the number of tokens (information pieces) to run in real time, but existing methods let one modality's importance scores decide what to keep in the other. This paper shows that for the same question, the moments that matter in audio and in video are often different, and proposes OmniScope, which lets each modality independently judge its own relevance to the query instead of one guiding the other. Without any additional training, it achieved the best average accuracy across compression settings on four benchmarks and two model sizes.
METAL MEDIA explanatory visual
OmniScope's three-stage compression pipeline
Evidence statusMeasured results reported
- Query-aware scoringUsing the same question as a reference, video relevance is scored independently via CLIP and audio relevance via the model's own internal embeddings, then a token budget is split accordingly
- Video pruning (AD-STC)Even-numbered frames keep overall scene information while odd-numbered frames keep only what changed from the previous frame, preserving both global context and temporal change
- Audio mergingSimilar audio tokens within each one-second window are paired and averaged together, reducing redundancy while keeping temporal continuity
- Performance validationAccuracy, speed, and memory were measured while retaining only 25-45% of tokens across four benchmarks and two model sizes, showing better results than competing methods
What they did
- Problem: existing omnimodal token compression methods use one modality's (often audio's) importance scores to decide what to keep in the other (video), but for the same question the moments that matter in audio and video often peak at different times, so this approach can discard information the answer actually needs.
- Method: OmniScope uses the query as a shared anchor but lets audio and video independently compute their own relevance scores and split the token budget accordingly. On the video side, an anchor-delta strategy (AD-STC) alternates between frames that preserve overall scene context and frames that preserve only what changed over time; on the audio side, similar tokens within each one-second window are merged to reduce redundancy while keeping temporal continuity.
- Testing: evaluated on four audio-video understanding benchmarks (WorldSense, DailyOmni, OmniVideoBench, Video-MME) and two sizes of the Qwen2.5-Omni model (7B and 3B) across multiple compression ratios.
- Result: keeping only 45% of tokens caused almost no accuracy loss, and even at an aggressive 25% retention the 7B model's average accuracy dropped only 0.35 points versus using all tokens, compared to a 1.55-point drop for the competing OmniZip method. It also achieved up to 3.53x faster prefill (initial input processing) and more than 15% GPU memory savings.
- Analysis: compression strengthens attention exchange across time windows and modalities; when one modality wrongly guides the other, this strengthened exchange gets steered toward answer-irrelevant cues, whereas letting each modality compress independently directs it toward genuinely query-relevant cues, as shown by attention visualizations.
Findings
- At the aggressive 25% token retention, the 7B model's average accuracy dropped only 0.35 points versus the full-token baseline, compared to drops of 1.55 points for OmniZip, 0.82 for FastV, and 1.57 for random pruning.
- At 45% retention, both the 7B and 3B models showed slight gains (+0.30, +0.03) or near-identical accuracy compared to using all tokens.
- On the 7B model at 25% retention, prefill time dropped from 6299ms to 1784ms (3.53x speedup) and peak GPU memory dropped from 28.31G to 24.00G, over 15% reduction.
- When independently sweeping video and audio compression ratios (Fig. 4), OmniScope showed the most gradual accuracy decline compared to all competing methods.
- Replacing the query-based independent scoring with uniform compression on both modalities caused a 1.65-point average drop, and letting one modality guide the other caused roughly a 1.0-point drop, confirming the value of independent per-modality compression.
Where it can be used
- Improving processing speed and memory use for services that must analyze audio and video together in real time, such as live video calls or streaming
- Reducing inference cost for existing omnimodal models without retraining, for tasks like long-video question answering, content moderation, or video captioning
- As a design reference for running audio-video models efficiently on resource-constrained (edge) devices
Limits and open work
- Experiments are limited to the Qwen2.5-Omni model family (7B, 3B) and four specific benchmarks; behavior on other omnimodal model architectures is untested.
- Results are measured with video input capped at 128 frames, so performance on much longer videos or other data conditions has not been separately verified.
- Video-side scoring requires running a separate external CLIP model, adding a fixed extra cost that can be relatively larger for short-answer generation tasks.
- The training-based competing method OmniSIFT could not be directly compared because its code was not publicly available at submission time.
- The authors note that the model's internal audio-based scoring approach is not yet reliable enough to apply to the video side as well, so video scoring still depends on an external CLIP model.
Why it matters
Services that must understand sound and video together in real time, such as live video calls, video search, or content moderation, are limited by the growing number of tokens to process; OmniScope offers a practical, training-free design principle to reduce this cost. Because it does not rely on model-specific fine-tuning, the approach could be worth trying with other omnimodal models facing similar efficiency constraints.
Terms in this paper
- Omnimodal large language model (OmniLLM) · an AI model that processes video, audio, and text together
- Token compression · a technique for reducing the number of information pieces (tokens) a model must process, saving speed and memory
- Query · the user's question, used here as the shared reference for deciding what to keep during compression
- Anchor-Delta compression (AD-STC) · a video compression scheme where some frames keep overall scene information (anchor) and others keep only what changed (delta)
- Prefill · the stage where the model processes the entire input once before generating an answer
Figures we cannot republish
- Figure 6
- Figure 7
- Figure 8
- Figure 9
Original abstract (English)
Existing token compression methods for omnimodal large language models typically rely on one modality to determine what to retain in the other. We show that this assumption often breaks down: for the same query, audio and video relevance often peaks at different moments. This cross-modal salience mismatch makes unidirectional guidance prone to discarding answer-critical cues under aggressive compression. We propose OmniScope, a training-free token compression framework that uses the query as a s
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctly
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
Latest from METAL MEDIA
Figures: Jinsen Su et al., arXiv:2607.23193, arxiv-nonexclusive