Evaluating Music Context Preservation: A Multi-facet Framework for Music Editing Systems
A new benchmark checks whether AI music editors accidentally wreck the parts of a song you didn't want changed
Music editing AI can change one thing (like genre or timbre) but often messes up rhythm, melody, or structure without meaning to. This paper builds MuseCPBench, the first standardized benchmark to measure this 'Music Context Preservation' across four musical facets. Testing five popular editing systems shows none of them reliably preserve everything, and each has different weak spots.
METAL MEDIA explanatory visual
A new benchmark checks whether AI music editors accidentally wreck the parts of a song you didn't want changed
- 01Problem: existing music editing systems (e.g. MusiConGen, InstructME, MS-Diffusion, ZETA) each check different, inconsistent things when claiming they preserve the original music, making fair comparison impossible.
- 02Solution: MuseCPBench organizes what should stay unchanged into four categories - harmony, rhythm & meter, structural form, and melodic content & motifs - with specific measurable metrics for each.
- 03Method: five representative editing systems (MusicGen, MusiConGen, MusicMagus, ZETA, RefinPaint) were tested on real music/MIDI datasets, comparing edited output against the original using these metrics, plus a 'Vanilla' baseline that generates music without any editing instruction.
- 04Result: RefinPaint (which refines existing MIDI rather than generating from scratch) preserved context best overall. MusicMagus kept tonal color well but nearly destroyed rhythm. MusicGen struggled most with timing/rhythm. ZETA kept rhythm well but lost melodic motifs.
- 05Finding: strong performance on one musical facet doesn't guarantee strong performance on another - no single system excelled everywhere, revealing a general blind spot in current music editing AI.
What they did
- Problem: existing music editing systems (e.g. MusiConGen, InstructME, MS-Diffusion, ZETA) each check different, inconsistent things when claiming they preserve the original music, making fair comparison impossible.
- Solution: MuseCPBench organizes what should stay unchanged into four categories - harmony, rhythm & meter, structural form, and melodic content & motifs - with specific measurable metrics for each.
- Method: five representative editing systems (MusicGen, MusiConGen, MusicMagus, ZETA, RefinPaint) were tested on real music/MIDI datasets, comparing edited output against the original using these metrics, plus a 'Vanilla' baseline that generates music without any editing instruction.
- Result: RefinPaint (which refines existing MIDI rather than generating from scratch) preserved context best overall. MusicMagus kept tonal color well but nearly destroyed rhythm. MusicGen struggled most with timing/rhythm. ZETA kept rhythm well but lost melodic motifs.
- Finding: strong performance on one musical facet doesn't guarantee strong performance on another - no single system excelled everywhere, revealing a general blind spot in current music editing AI.
| Method | Backbone | Task | Edit | MCP Evaluation? | Facets |
|---|---|---|---|---|---|
| MusicGen [3] | Transformer | Text / Music-conditioned generation | — | ✗ | — |
| MusiConGen [8] | Transformer | Text / Music-conditioned generation | Harmony, Rhythm | ✓ | Harmony, Rhythm |
| AUDIT [18] | Diffusion | Instruction-guided editing | Local attributes | ✗ | — |
| InstructME [6] | Diffusion | Instruction-guided editing | Harmony, Rhythm | ✓ | Harmony, Rhythm |
| MusicMagus [20] | Diffusion | Music inversion | Global structure | ✗ | — |
| RefinPaint [15] | Diffusion | Iterative inpainting | Local segments | ✗ | — |
| MS-Diffusion [11] | Diffusion | Joint generation and separation | Stems (sources) | ✓ | Stem Fidelity |
| ZETA [10] | Diffusion | Editing and inversion | Structure | ✓ | Structure (human-judged) |
| Baseline | Harmony & Tonality | Rhythm & Meter | ||||
|---|---|---|---|---|---|---|
| Circle of Fifth-distance ↓ | Chroma DTW Similarity ↑ | Major Minor Score ↑ | Δ BPM ↓ | Beat F-measure ↑ | Information Gain ↑ | |
| Vanilla | 0.333 | 0.860 | 0.188 | 26.638 | 0.229 | 0.141 |
| MusicGen | 0.197 | 0.870 | 0.334 | 36.690 | 0.273 | 0.391 |
| MusiConGen | 0.210 | 0.902 | 0.474 | 0.000 | 0.492 | 0.362 |
| MusicMagus | 0.068 | 0.955 | 0.738 | 13.262 | 0.030 | 0.029 |
| ZETA | 0.177 | 0.945 | 0.441 | 4.125 | 0.732 | 0.604 |
| RefinPaint | 0.009 | 0.988 | 0.943 | 8.825 | 0.949 | 0.951 |
Why it matters
For anyone building or using AI music editing tools, this gives a concrete, comparable way to check if an edit tool ruins the rhythm or melody while changing style - something that was previously unmeasured. It also gives model builders a diagnostic map of exactly where their architecture (autoregressive vs diffusion vs refinement-based) tends to fail.
Terms in this paper
- Music Context Preservation (MCP) · the ability of a music editing AI to keep the parts of a song unchanged that weren't meant to be edited
- MuseCPBench · the benchmark introduced here to measure MCP consistently across systems
- harmony / rhythm & meter / structural form / melodic content & motifs · the four musical dimensions the benchmark checks are preserved after editing
- circle-of-fifths distance · a music-theory based measure of how close two musical keys are to each other
- Beat F-measure / Information Gain · metrics for how well the timing of beats in edited music matches the original
- Adjusted Rand Index (ARI) · a score measuring how well two segmentations of a song's sections agree with each other
- autoregressive generation · generating audio step-by-step in sequence, which can drift or accumulate errors over longer pieces
Original abstract (English)
Music editing plays a vital role in modern music production, with applications in film, broadcasting, and game development. Recent advances in music editing systems have enabled diverse editing tasks such as timbre transfer, instrument substitution, and genre transformation. However, many existing works overlook evaluating their ability to preserve musical facets that should remain unchanged during editing, which we define as Music Context Preservation (MuseCP). While some studies do consider MuseCP, their evaluation protocols and metrics are not comprehensive. To address this, we introduce the first MuseCP evaluation framework, MuseCPEval, that covers four categories of music facets with fine-grained and well-tailored metrics to capture nuanced changes in music attributes. Objective validation and a human study demonstrate the effectiveness of these metrics. Moreover, the case studies on diverse music editing systems illustrate the practical utility of these metrics as a testbed and diagnostic tool, providing insights into the strengths and limitations of existing systems. We hope our metrics and findings can offer practical guidance for developing more effective and reliable music editing strategies with strong MuseCP capability
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- Beyond Imitation: Filtering On-Policy Distillation by Reasoning ProgressA fix for AI models that get penalized by their teacher even when they're reasoning correctly
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
Latest from METAL MEDIA
Figures: Yash Vishe et al., arXiv:2512.14629, CC BY 4.0