Specification-delta-driven data governance: an empirical study of the «spec-delta» as the unit of change in lakehouse data platforms
Should data-platform changes be reviewed as a 'spec-delta' instead of a code diff? A study design, not results yet
This paper proposes treating contractual changes in lakehouse data platforms -- new datasets, metric redefinitions, SLA changes, access policy edits -- as a reviewable 'spec-delta' (a requirements increment) rather than a plain code pull request. It lays out a taxonomy of which change types should benefit most, an eight-task experimental bank, response variables to measure, and a statistical analysis plan, but no experiment has actually been run yet. The authors are explicit that the contribution is the reproducible study design and applicability guide, not a tool or measured results.
METAL MEDIA explanatory visual
How the proposed spec-delta experiment would flow
Evidence statusProposal only · no measured results yet
- A change arisesOne of eight task types (T1-T8) occurs, e.g. registering a new dataset, redefining a metric, or changing an SLA
- Reviewed under two conditionsThe same participant completes an equivalent task under C1 (conventional code PR) and C2 (spec-delta), order counterbalanced
- Promotion gate checkPassing quality tests, emitted lineage, assigned owner, and computed downstream impact are required before promotion to Silver or Gold
- Four variables measuredDiscovery-to-deployment time, defect density in Silver/Gold, cross-BI-tool metric divergence, and NASA-TLX cognitive load are recorded
- Analysis (planned)Linear mixed models and Poisson regression are planned for analysis, but the results tables remain empty templates awaiting real data
What they did
- Spec-Driven Development already established that specifications, not code, should govern AI-assisted software work, and OpenSpec introduced the idea that each change should produce a reviewable requirements increment. This paper is the first to formalize that idea specifically for data platforms.
- The authors build a taxonomy distinguishing 'contractual' changes (new dataset registration, metric semantics, SLA/SLO changes, access policy changes) from 'code' changes (internal refactors, performance tuning), hypothesizing that spec-deltas help most for the former and least for the latter.
- They design a within-subject crossover experiment comparing a spec-delta workflow (C2) against a conventional code pull-request workflow (C1), with each participant doing equivalent tasks under both conditions in counterbalanced order.
- Planned response variables are discovery-to-deployment time, defect density reaching the Silver and Gold layers of the medallion lakehouse architecture, cross-tool metric divergence between BI tools, and reviewer cognitive load measured with NASA-TLX.
- A reference lab setup on Azure Databricks with a Global Superstore dataset, eight paired change tasks (T1-T8), empty results-table templates, and a fully worked example spec-delta for redefining a profit-margin metric are all provided, but the laboratory itself has not yet been executed.
Findings
- No measured results have been reported yet.
Where it can be used
- Documenting high-impact data-platform changes -- new datasets, metric redefinitions, SLA edits, access policy changes -- as reviewable requirement increments instead of bare code pull requests
- Designing promotion gates that require passed quality tests, emitted lineage, an assigned owner, and computed downstream impact before data is promoted to Silver or Gold layers
- Centralizing metric definitions in a single semantic layer to prevent the same metric (e.g. revenue, churn) from being computed differently across BI tools like Power BI and Fabric
Limits and open work
- Only a study design and taxonomy are presented; no participants have run the experiment yet and no outcome numbers exist.
- The eight example tasks and taxonomy are built around a specific reference setup (Azure Databricks, Global Superstore dataset), so transferability to other organizations or lakehouse engines is untested.
- The claim that contractual changes benefit more than code-only changes is a stated hypothesis to be tested, not a confirmed finding.
- Future work explicitly includes running the actual laboratory, cross-validating the taxonomy across multiple organizations and engines, and studying language-model-assisted spec-delta writing under human review.
Why it matters
Teams often treat data-pipeline changes like ordinary code reviews even when the real risk is a broken contract -- a metric definition, an access rule, an SLA -- that downstream consumers depend on. This work offers a concrete, testable way to make those contractual changes legible to reviewers and a rigorous plan to check, without overclaiming, whether that actually saves time or reduces defects.
Terms in this paper
- spec-delta · A small, reviewable increment of specification describing just what changes, rather than a whole document or the code itself
- medallion architecture · A lakehouse data structure with three quality tiers: raw Bronze, cleaned Silver, and analytics-ready Gold
- data contract · A formal, runtime-enforced agreement between data producers and consumers about schema, meaning, and quality
- NASA-TLX · A standard questionnaire for measuring how mentally demanding a task felt to the person who performed it
- GIVEN/WHEN/THEN scenario · A way of writing testable acceptance criteria: given a condition, when an action happens, then an observable outcome must follow
Figures we cannot republish
- Figure 0
Original abstract (English)
Spec Driven Development SDD has consolidated the idea that the specification rather than the code should be the primary artefact governing AI assisted work. Tools such as GitHub Spec Kit, and proposals such as Constitutional SDD, have formalised this principle in the software domain, while the executable data-contracts literature has extended it to schema and quality enforcement at run time. Nevertheless, the treatment of the specification delta OpenSpec's core idea that every change should produce a reviewable increment of requirements as the unit of change in data platforms remains empirically unexplored, even though many data-platform changes are contractual (new datasets, service-level agreements, metric semantics, access policies) rather than purely code changes. This work formalises the spec-delta concept, proposes a taxonomy of data platform changes according to their suitability for incremental specification, and defines a controlled experiment comparing a spec-delta-driven workflow against a conventional code pull-request workflow without a delta. The response variables are discovery to deployment time, the density of defects reaching the Silver and Gold lakehouse layers, cross-tool metric divergence, and reviewer cognitive load measured with NASA TLX. The paper explicitly reserves a demonstration-and-laboratory section for instantiation on a real lakehouse environment. The contribution is not a tool but reproducible evidence and an applicability guide that helps to avoid the up front over specification antipattern.
Read on arXivLatest papers
- SWE-bench Science: Can Coding Agents Resolve Engineering Tasks in Science?AI coding agents were tested on fixing real scientific software, and even the best one failed more than half the time
- FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM ServingMaking sparse attention fast enough and accurate enough for real LLM serving, not just papers
- PolicyGuide: From Guarding One Action to Guiding the Whole Workflow for Policy-Compliant LLM AgentsMaking customer-service AI agents follow the whole procedure, not just avoid one bad action
- EXIMO: VLM Guided Exploration of VLA PoliciesTeaching a robot new chores without human teleoperation, by letting a chatty AI supervise it
- EnvHarness: Awakening Static Worlds for Agent LearningInstead of building new training worlds from scratch, this work adds a plug-in layer that reshapes existing ones around each agent's actual weaknesses
- Bounded Sovereignty and the Control Tax: Pricing AI Oversight When the Deployer Does Not Own the ModelCompanies that rent AI instead of owning it can only do half of AI safety oversight
- PersonalBench: Measuring the Authorship Gap in LLM PersonalizationAI can be prompted to write 'like someone,' but its own voice never fully disappears
- Automated Summarization of Financial News Using Large Language Models and Retrieval-Augmented Generation: An Early Empirical Study (Fall 2023)Testing AI summaries of stock news, the simple approach beat the trendy retrieval-based one