Air Traffic Control Using Large Language Models: Prompt Engineering, Architecture, and Evaluation

arXiv:2608.192992026-08-21

研究者让AI扮演空中交通管制员,结果发现它说话像模像样,但关键指令却经常出错

研究团队以旧金山湾区一次通用航空飞行的真实塔台通话手工转录记录作为标准答案,测试了九个大语言模型在完整36轮对话中扮演管制员的表现。结果显示,提示词写得越详细、规则越多,表现反而越差;而当模型不断参考自己此前的回复时,早期的小错误会在对话中不断累积放大。最终这些模型很擅长模仿管制员的说话方式,却难以做出真正正确的运行判断。

METAL MEDIA 解读图

研究者让AI扮演空中交通管制员,结果发现它说话像模像样,但关键指令却经常出错

01研究使用一段真实的36轮飞行员-管制员通话记录作为标准答案,让九个模型(六个开源、三个闭源)在不断累积的对话历史条件下生成管制员回复,模拟真实部署场景
02研究设计了从极简角色设定到规则繁多的五种提示词,结果最简单的提示词表现最好,而规则最严苛的提示词相似度反而下降了35%
03当模型依赖自己此前的回复继续对话时,早期错误(比如反复只回答'收到')会随对话推进不断累积、表现持续下滑;但若换成真实管制员的历史记录,这一崩溃现象就被修复
04在提示词中加入一段完整的示例对话(上下文学习)能提高与标准答案的表面相似度,但用GPT-5.5作为评判模型打分时,并未提升实际运行判断的正确性
05模型在呼号处理和简洁性方面平均得分高达5分中的4.1分,但在运行正确性上仅得2.1分,而且开源模型规模越大也未能缩小这一差距

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

研究使用一段真实的36轮飞行员-管制员通话记录作为标准答案,让九个模型(六个开源、三个闭源)在不断累积的对话历史条件下生成管制员回复,模拟真实部署场景
研究设计了从极简角色设定到规则繁多的五种提示词,结果最简单的提示词表现最好,而规则最严苛的提示词相似度反而下降了35%
当模型依赖自己此前的回复继续对话时,早期错误(比如反复只回答'收到')会随对话推进不断累积、表现持续下滑;但若换成真实管制员的历史记录,这一崩溃现象就被修复
在提示词中加入一段完整的示例对话(上下文学习)能提高与标准答案的表面相似度,但用GPT-5.5作为评判模型打分时,并未提升实际运行判断的正确性
模型在呼号处理和简洁性方面平均得分高达5分中的4.1分,但在运行正确性上仅得2.1分,而且开源模型规模越大也未能缩小这一差距

Figure 1: Overall architecture. (1) Generation: pilot transmissions transcribed from an experimental Bay Area tour flight are issued one turn at a time to the language model, which is conditioned on a system prompt and on the communication history accumulated so far; each pseudo-ATC response is appended back into that history. (2) Teacher-forced generation: the same scenario is replayed with the controller side of the history replaced by the ground truth transmissions. The example contrasts the two regimes at turn 24. Conditioned on its own replies (left), the model has fallen into an Acknowledgment loop and answers “Roger,” missing the required radar-service termination and Hayward handoff. Given the true history (right), the same turn is answered with the correct facility, the correct “keep the code” instruction, and the correct frequency 120.2. (3) Evaluation: every turn is scored by automatic similarity metrics, by an LLM-as-judge, and—on a stratified subsample—by a human expert.

Figure 2: Prompt × (ICL × history) interaction, pooled over the nine models (mean ± 95% CI across models). Color encodes the dialogue history the model conditions on, line style encodes whether a worked in-context example was supplied. Under the light prompts C1–C3 the two history regimes are indistinguishable; under the over-specified C5 the self-history condition collapses and ground-truth grounding recovers it.

Table 1: Per-prompt similarity to the C0 ground truth, by condition (ICL × history), averaged over the 9 models and 3 attempts. Best per column in bold.
	ROUGE-L	Semantic cosine
Prompt	no ICL, self	no ICL, GT	ICL, self	ICL, GT	no ICL, self	no ICL, GT	ICL, self	ICL, GT
C1	0.216	0.209	0.244	0.244	0.464	0.449	0.497	0.490
C2	0.217	0.214	0.243	0.230	0.482	0.472	0.492	0.477
C3	0.219	0.212	0.243	0.224	0.487	0.470	0.493	0.467
C4	0.209	0.220	0.216	0.238	0.430	0.445	0.447	0.470
C5	0.167	0.218	0.159	0.224	0.302	0.383	0.298	0.396

Figure 3: Cumulative running-mean similarity over the 36-turn conversation (ICL on, pooled over models). Under C1–C3 the self- and ground-truth-history trajectories track each other; under C5 self-history drifts steadily downward as early errors propagate through the dialogue history, while ground-truth injection holds the trajectory flat.

Figure 4: LLM-judge score per rubric dimension, pooled over the nine models. Color encodes the dialogue history the model conditions on; hatching marks the in-context learning condition. Models score highly on callsign handling and conciseness but poorly on operational correctness.

Table 2: Per-model similarity to the C0 ground truth, by condition (ICL × history), averaged over prompts C1–C5 and 3 attempts. Best per column in bold.
	ROUGE-L	Semantic cosine
Model	no ICL, self	no ICL, GT	ICL, self	ICL, GT	no ICL, self	no ICL, GT	ICL, self	ICL, GT
Open-source
Qwen2.5-7B	0.197	0.215	0.212	0.234	0.413	0.458	0.443	0.483
Llama-3.1-8B	0.193	0.181	0.208	0.203	0.437	0.403	0.452	0.418
Gemma-2-9B	0.188	0.218	0.216	0.248	0.405	0.441	0.438	0.487
Qwen2.5-14B	0.203	0.200	0.215	0.195	0.481	0.464	0.483	0.456
Qwen2.5-32B	0.215	0.213	0.240	0.234	0.460	0.453	0.476	0.474
Mixtral-8x7B	0.123	0.149	0.116	0.150	0.336	0.361	0.310	0.373
Average	0.186	0.196	0.201	0.211	0.422	0.430	0.434	0.449
Closed-source
GPT-5.4	0.246	0.236	0.255	0.247	0.470	0.450	0.470	0.449
GPT-5.4-mini	0.246	0.237	0.253	0.267	0.447	0.469	0.463	0.482
Claude-Sonnet-4.6	0.240	0.282	0.275	0.310	0.447	0.495	0.474	0.519
Average	0.244	0.252	0.261	0.275	0.455	0.471	0.469	0.483

Figure 5: Per-dimension scores from the LLM judge (a) and the human expert (b) on the same 180 annotated outputs, broken out by condition (color = dialogue history, hatching = in-context example). Both raters produce the same cross-dimension profile, peaking on callsign handling and conciseness and dipping on operational correctness, but the human sits about one point higher on every dimension. Because both panels use the identical 180-sample subset, the vertical offset between them is the genuine human–judge leniency gap.

Table 3: Hypothesis tests for the three experimental contrasts, computed on per-model means across the nine models. Paired contrasts use an exact Wilcoxon signed-rank test (W); the closed-versus-open contrast uses a one-sided Mann–Whitney U test. Δ is reported in the direction named in each group heading, wins counts models favoring that direction, and the last column is the effect size (dz for the paired contrasts, Cohen’s d for the unpaired one). Bold indicates p<0.05.
Measure	Δ	Wins	Statistic	p	Effect
In-context example vs. none (paired, W, n=9)
ROUGE-L (pooled)	0.016	8/9	1.0	0.008	1.46
Normalized BLEU (pooled)	0.009	8/9	1.0	0.008	1.47
LLM-judge overall	0.036	5/9	13.0	0.547	0.25
Ground-truth vs. self dialogue history (paired, W, n=9)
ROUGE-L (pooled)	0.010	5/9	14.0	0.359	0.49
BLEU-1 (pooled)	0.044	9/9	0.0	0.004	1.98
ROUGE-L (prompt P1)	−0.003	4/9	20.0	0.820	−0.11
ROUGE-L (prompt P5)	0.057	9/9	0.0	0.004	1.72
LLM-judge overall	−0.078	3/9	14.5	0.371	−0.40
Closed- vs. open-source (unpaired, U, 3 vs. 6)
ROUGE-L	0.059	—	18.0	0.012	2.03
LLM-judge overall	1.050	—	18.0	0.012	4.43

Figure 6: Model-level agreement between each automatic similarity metric and the LLM judge. Each point is one model, plotted as its mean metric score against its mean judge rating, with a fitted line and 95% confidence band. ROUGE-L and TF-IDF track expert judgment closely; BLEU-1 and sentence-embedding similarity do not. The open/closed legend in (a) applies to all four panels.

Table 4: LLM-judge scores (1–5) per model, averaged over conditions and the top-25 judged outputs per cell. Best per column in bold.
Model	Phraseology	Correctness	Callsign	Conciseness	Safety	Halluc.-free	Overall
Open-source
Qwen2.5-7B	2.46	1.75	4.36	3.97	2.63	2.68	1.99
Llama-3.1-8B	2.40	1.74	3.65	3.88	2.51	2.22	1.89
Gemma-2-9B	2.80	1.94	4.14	4.34	2.80	2.37	2.12
Qwen2.5-14B	2.35	1.70	4.44	3.17	2.47	2.00	1.84
Qwen2.5-32B	2.81	2.00	4.29	3.83	2.83	2.33	2.20
Mixtral-8x7B	2.00	1.32	3.54	3.42	2.11	1.82	1.45
Average	2.47	1.74	4.07	3.77	2.56	2.24	1.91
Closed-source
GPT-5.4	3.84	2.86	4.24	4.78	3.81	3.77	3.13
GPT-5.4-mini	3.59	2.57	4.15	4.61	3.54	3.20	2.82
Claude-Sonnet-4.6	3.72	2.68	4.35	4.75	3.60	3.31	2.95
Average	3.71	2.71	4.25	4.71	3.65	3.43	2.96

Table 5: Spearman rank-correlation matrix over all six evaluation measures, computed on the 180 human-annotated outputs so that every measure is available for every observation. The two reference-overlap metrics (ROUGE-L, TF-IDF) are the only automatic measures that track either human-facing judgement.
	ROUGE-L	TF-IDF	BLEU-1	Semantic	LLM judge
TF-IDF	0.73
BLEU-1	0.58	0.36
Semantic	0.32	0.47	0.30
LLM judge	0.53	0.49	0.02	0.10
Human	0.34	0.38	-0.05	0.09	0.67

Figure 7: Mean overall rating assigned by the LLM judge and by the human expert, split by the judge’s own score band (mean ± 95% CI). The human is systematically more lenient, and the gap grows as output quality falls.

Figure 8: Per-model similarity by history mode (ICL on), grouped into open- and closed-source models. Bars are self history (hatched //) versus ground-truth history (hatched \\); panels are the four similarity metrics.

Table 6: Agreement between the three evaluation layers. Sample-level statistics treat each scored output as one observation; model-level statistics use per-model means (n=9). κw is quadratic-weighted Cohen’s kappa, defined only when both measures share the 1–5 ordinal scale.
Comparison	Level	Pearson r	Spearman ρ	κw
Automatic metric vs. LLM-judge overall (sample n=4,500)
ROUGE-L	sample	0.51	0.52	–
ROUGE-L	model	0.90	0.93	–
TF-IDF	sample	0.50	0.48	–
TF-IDF	model	0.91	0.90	–
BLEU-1	sample	0.12	0.06	–
BLEU-1	model	0.43	0.22	–
Semantic	sample	0.08	0.09	–
Semantic	model	0.67	0.50	–
Automatic metric vs. human overall (sample n=180)
ROUGE-L	sample	0.31	0.34	–
ROUGE-L	model	0.73	0.81	–
TF-IDF	sample	0.32	0.38	–
TF-IDF	model	0.72	0.79	–
BLEU-1	sample	0.02	-0.05	–
BLEU-1	model	0.17	-0.03	–
Semantic	sample	0.09	0.09	–
Semantic	model	0.39	0.28	–
LLM judge vs. human annotator (sample n=180)
Overall rating	sample	0.71	0.67	0.60
Overall rating	model	0.69	0.86	–

Table 7: Per-dimension and per-stratum agreement between the LLM judge and the human annotator (n=180). Δ is the human mean minus the judge mean; positive values mean the human is more lenient. Within-stratum correlations are attenuated by range restriction and κw is uninformative there, so it is reported only for the pooled sample.
Dimension	Judge mean	Human mean	Δ	Spearman ρ	κw
Phraseology	3.24	4.30	+1.06	0.45	0.36
Correctness	2.89	3.79	+0.90	0.63	0.52
Callsign	4.43	4.86	+0.43	0.35	0.43
Conciseness	4.34	4.51	+0.17	0.41	0.59
Safety	3.63	3.91	+0.28	0.53	0.59
Halluc.-free	3.61	3.96	+0.34	0.48	0.46
Overall	3.12	3.91	+0.79	0.67	0.60
Overall rating, split by judge stratum
Low (n=45)	1.13	2.58	+1.44	0.44	–
Mid (n=45)	2.87	3.78	+0.91	0.36	–
High (n=90)	4.24	4.64	+0.40	0.20	–

为什么重要

在业界探索将AI用于空管这类事关安全的关键任务之际,这项研究用具体数据揭示了当前大语言模型在说得像与做得对之间的明显落差。这提示AI更适合承担辅助性角色,如起草常规通话、支持培训模拟或筛查录音,而非直接进入管制决策环节。

本文术语

LLM评判(LLM-as-judge) · 让另一个大语言模型代替人工为生成结果打分的评估方式
上下文学习(ICL) · 在提示词中直接给模型提供一个示例对话供其模仿
ROUGE-L、BLEU、TF-IDF · 衡量生成文本与参考文本在用词和顺序上相似程度的自动评估指标
教师强制(teacher forcing) · 生成下一轮回复时,用真实正确的历史记录替换模型自己此前的回复
二次加权科恩卡帕系数(κw) · 衡量两个评分者在等级量表上打分一致程度的统计指标

论文原文摘要（英文）

Air traffic control (ATC) communication is a safety-critical dialogue that remains largely human-driven even as other parts of air traffic management have been semi-automated. In this article, we experimentally evaluate whether large language models (LLMs) can generate operationally realistic ATC transmissions. An experimental general-aviation flight flying over the San Francisco "Bay Tour" route is hand-transcribed and used as ground truth (P0). Through a pilot-in-the-loop process we design five prompt structures (P1-P5) of increasing constraint and embed them in a stateful multi-turn pipeline, where the model plays ATC to a fixed pilot transcript while conditioning on the accumulating dialogue history. Across nine open- and closed-source LLMs we vary the prompt, the presence of a worked transcript from a different experimental flight as an in-context example, and whether the model conditions on its own prior replies or on injected ground-truth history. Turns are scored with lexical, structural, and semantic similarity metrics and by an LLM-as-judge (GPT-5.5) validated against human expert annotation. Supplying a worked example improves similarity, but tightening the prompt does not: the lightest prompts perform best and the most heavily scripted one collapses as its own errors accumulate through the dialogue, which injecting correct history repairs. These results outline a concrete path and its current limits toward LLM-assisted ATC.

作者 · Mahyar Ghazanfari, Matthias Casanova, Jordan Kam, Alex Zongo, Peng Wei, Torsten Darrell, Alexandre Bayen

在 arXiv 阅读

METAL MEDIA 最新报道

图片来源： Mahyar Ghazanfari et al., arXiv:2608.19299, arxiv-nonexclusive

Air Traffic Control Using Large Language Models: Prompt Engineering, Architecture, and Evaluation

研究者让AI扮演空中交通管制员,结果发现它说话像模像样,但关键指令却经常出错

他们做了什么

为什么重要

本文术语

最新论文

METAL MEDIA 最新报道