Table 1: Relationship between ACR and measured FLOPs Ratios. ACR is a selected-window routing diagnostic. Measured FLOPs Ratios require matched profiler counters and are not inferred from selected windows.
Metric
What it measures
Source in this paper
Claim supported
ACR
selected attention-window coverage under rectangular query-by-window accounting
AAH policy logs and final branch-usage frequencies
head-wise context-allocation structure
Measured Attention FLOPs Ratio
profiler GPU FP operations inside attention ranges divided by the matched pure-attention baseline
Nsight / profiler suite only
hardware-measured attention operation count, when available
Measured Total FLOPs Ratio
profiler GPU FP operations for the full profiled forward or step divided by the matched pure-attention baseline
Nsight / profiler suite only
hardware-measured end-to-end operation count, when available
Table 2: Main 1B / 4096-token protocol fields and W&B row-selection rule. Additional artifact-packaging metadata is listed in Appendix A.5.
Protocol field
Value used for the main 1B / 4096-token suite
Principal regimes
Full attention, Grouping off, Full adaptive, Shallow freeze, and Deep practical reuse.
Intended controlled fields
Tokenizer, data pipeline, optimizer schedule, batch construction, model width/depth, seed, and Transformer output interface are held fixed by local experiment configuration across the five paper-facing regimes.
Exported metadata
The W&B export records per-regime run names, config paths and hashes, run seeds, checkpoint step, and git commit IDs. Appendix A.5 lists the remaining artifact-packaging fields to include in a public release bundle.
Precision
bf16 in the W&B run configuration.
Batch size
1.
Context length
4096 tokens.
Candidate windows
[512,1024,2048,4096].
Control/EMA settings
Control updates every five steps; feature EMA coefficient 0.9; final resolution EMA coefficient 0.15.
Checkpoint rule
Save checkpoints at optimizer steps 1000, 5000, and 10000; paper tables use the 10000-step final checkpoint.
W&B row-selection rule
For Full attention rows, set ACR to 1.0000 by definition when AAH-specific ACR fields are absent. For AAH rows, select completed seed-0 rows with final checkpoint step 10000 and non-null validation, ACR or branch-usage data, throughput, and peak allocated memory fields when available; ignore stale summaries that conflict with wandb_results_new/.
Dense backend status
The main tables report ACR as a routing diagnostic, not as a FLOPs Ratio. The dense-masked backend has not established skipped query–key work; therefore these rows omit backend work-accounting and measured FLOPs Ratio columns.
Table 3: Configuration differences among the main 1B / 4096-token regimes. Performance metrics are reported separately in Tables 4 and 6.
Method
Grouping
Hierarchy
Joint scorer
Full attention
-
-
-
Grouping off
off
-
off
Full adaptive
feature-derived adaptive
[2,2,2,2] adaptive
wide joint
Shallow freeze
feature-derived frozen
[2]
wide joint
Deep practical reuse
cached level-0
[2,2,2,2] reuse
wide joint
Table 4: Main 1B / 4096-token training comparison for the Full attention reference and principal AAH-v3 execution regimes. ACR is the selected attention-coverage policy proxy. The dense-masked path does not establish skipped backend work, so backend work accounting is omitted. Memory is peak allocated GPU memory from perf/gpu_alloc_max_mb/1024. Bold indicates the best value in each metric column.
Method
Val. loss
ACR
Token/s
Memory (GB)
Full attention
6.5672
1.0000
4447
35.18
Grouping off
6.5655
0.5814
3653
36.00
Full adaptive
6.5590
0.3066
3319
36.78
Shallow freeze
6.5367
0.3724
3479
36.77
Deep practical reuse
6.5549
0.2891
3381
36.62
Table 5: Phase 1 3000-step quality/structure screening ablation at 4096-token context. Lower validation loss is better. These short-budget rows test whether head-window assignment structure and routing stability matter; they are not a replacement for matched 10000-step paper-grade confirmation.
Phase 1 row
Val. loss
Conservative interpretation
Pure baseline
7.3226
uniform full-span reference for the screening budget
Shallow control interval 10
7.2775
best screening row; stable shallow routing is useful in this budget
Shallow freeze
7.2920
feature-derived shallow structure improves over pure baseline
Fixed random grouping
7.2831
strong control; hierarchy adaptivity alone is not proven causal
Fixed 1024
7.2834
strong fixed-window control; local context allocation itself is competitive
Shallow shuffle-post-select
7.3074
shuffling selected assignments weakens the shallow result
Full adaptive shuffle-post-select
7.3404
shuffling selected assignments is worse than the pure baseline
Table 6: Main 1B / 4096-token inference comparison for the Full attention reference and principal AAH-v3 execution regimes. Inference ACR is reconstructed from final inference branch-usage frequencies as ∑WfWW/T with T=4096. The dense-masked path does not establish skipped backend work; no measured FLOPs Ratio is reported without matched profiler GPU FP-operation totals. Bold indicates the best value in each metric column.
Method
Val. loss
Val. ppl
ACR
Token/s
Full attention
6.5598
706.16
1.0000
14425
Grouping off
6.5580
704.88
0.5793
4882
Full adaptive
6.5546
702.50
0.2484
4318
Shallow freeze
6.5293
684.93
0.3400
6983
Deep practical reuse
6.5514
700.24
0.3616
4595
Table 7: Systems diagnostics for the measured-GPU-FLOPs claim boundary. ACR is a routing diagnostic; measured FLOPs claims require Nsight-derived GPU floating-point-operation counters.
Evidence source
Current result
Claim boundary
Backend 4096 realized-attention suite
AAH rows expose selected-window ACR and backend span diagnostics under FlexAttention or FlashAttention execution paths.
Useful for routing/backend checks, but not a measured FLOPs metric.
Backend 4096 Nsight summary
FlashAttention AAH rows are about 1.59x–1.61x the pure FlashAttention measured GPU FLOPs.
Negative evidence for a GPU-FLOPs-reduction claim in the current backend implementation.
PRO6000 FLOPs-lab lower-overhead probes
Later probes approach but do not beat pure FlashAttention or pure dense MHA.
Overhead can be reduced, but measured GPU-FLOPs savings are not established.
Dense-framework FLOPs probes
Dense-framework variants approach a ratio of 1.0 but remain above the pure dense baseline.
Dense-framework probes also do not establish measured GPU-FLOPs savings.
Table 8: Planned appendix diagnostic runs for the fixed 1B / 4096 / 10000-step / seed-0 protocol. These rows define intended mechanism checks, not completed main-text evidence.
Appendix run
Variable changed
appendix_4096_control_off
disable controller decisions and use no adaptive control
appendix_4096_fixed_random_grouping
replace feature-derived grouping with fixed random grouping
appendix_4096_freeze_after_warmup_passthrough
freeze or pass through topology after warmup
appendix_4096_independent_scoring
use an independent fixed scorer instead of joint sibling scoring
appendix_4096_no_parent_constraint
disable parent index constraint
appendix_4096_no_feature_ema
disable feature EMA smoothing
appendix_4096_simple_policy
planned deterministic entropy/norm-based or random-window policy baseline
Table 9: Appendix-only internal pretrained compatibility smoke test on capped deterministic Qwen3-4B-Base subsets at 4096-token context. These are not official full benchmark scores and should not be cited as benchmark results or used for external model comparisons. Best score for each benchmark is bolded, with ties bolded.
Benchmark
Full attention
Full adaptive
Deep practical reuse
Shallow freeze
Grouping off
Language
MMLU
70.9
71.1
70.9
70.7
70.9
MMLU-Pro
28.5
27.7
27.7
27.7
28.5
GPQA-Diamond
34.8
34.3
34.3
33.8
35.4
ARC-Challenge
86.3
86.3
86.3
86.3
86.3
HellaSwag
44.3
44.3
44.3
44.3
44.5
TriviaQA
37.5
37.5
37.5
37.5
37.5
C-Eval
71.5
71.5
71.5
71.5
71.9
Math and code
GSM8K
13.3
13.3
13.3
14.8
13.3
HumanEval
34.4
34.4
34.4
34.4
34.4
MBPP
3.1
3.1
3.1
3.1
3.1
Table 10: Separate 1B-suite routing-diagnostic context for the AAH regimes used in the Qwen3 compatibility check. ACR is taken from the 1B final-checkpoint inference comparison in Table 6; it is not a Qwen3 downstream-run hardware or FLOPs measurement. Max absolute delta is the largest benchmark-score change, in percentage points, relative to the Full attention row in Table 9.
Standard multi-head attention (MHA) gives every head the same full causal context span, although heads can serve different contextual roles. Some heads may rely mainly on nearby lexical or syntactic context, while others may depend on longer-range relations such as entity interactions, discourse links, or state changes. We present Asymmetric Attention Heads (AAH), a head-wise context- allocation framework that treats context length as an explicit per-head or per-group allocation variable. AAH groups heads using feature-derived statistics, organizes these groups hierarchically, and assigns causal local windows while preserving the standard flat MHA output interface. In 4096- token seed-0 experiments, several AAH-style local-allocation variants achieve lower validation loss than pure full attention. Short-budget ablations show that stable local allocation and head-window assignment structure matter, while fixed/local controls can be competitive with adaptive hierarchy. We interpret AAH as a structured head-wise context-allocation mechanism for quality and analysis, with Attention Coverage Ratio (ACR) reported as a selected-window routing diagnostic
