Everything K-culture — comebacks to K-beauty, straight to your inboxGet it in your inbox›
SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation
arXiv:2608.185652026-08-18
AI-written factory control code needs to be actually run, not just compiled, to prove it works
SemaPLC is an agent harness that makes a large language model keep working on industrial control code until three external checks - a specification audit, compilation, and live runtime execution - all confirm success, instead of letting the model decide for itself when it's done. It was tested on 117 standalone code-generation tasks and 65 tasks that require the generated logic to compile and run inside real, existing plant projects, across seven different backbone models. While methods looked similar under static, text-only checks, running the code on a live controller revealed SemaPLC was far more reliable than existing baselines.
METAL MEDIA explanatory visual
AI-written factory control code needs to be actually run, not just compiled, to prove it works
01SemaPLC generates code for programmable logic controllers (PLCs), the industrial computers that run plants, and refuses to declare a task finished until logged external checks - a specification review, compilation, and execution on a live PLC runtime - all pass, rather than trusting the model's own judgment.
02It was evaluated on two tracks: 117 independent code-unit tasks from an existing benchmark, and 65 tasks built from ten real industrial plants where generated logic must integrate into and run inside an existing project.
03On the standalone-unit track, SemaPLC achieved the highest strict verified pass rate on all seven models (72.6% average), 17.3 points higher than the same setup with the harness stripped out.
04On the project-integration track, it had the best average compile success rate (89.4%); static, text-only checks showed only small gaps between methods, but when logic was actually deployed and its execution trace compared to a reference, SemaPLC scored 52.2 on average versus at most 31.4 for other methods.
05Its advantage narrowed on the strongest model tested (GPT-5.5), and because the runtime tests use a fixed, pre-derived set of scenarios, performance under unseen operating conditions remains unmeasured.
An explanatory diagram made by METAL MEDIA, not a figure supplied by the paper's authors.
What they did
SemaPLC generates code for programmable logic controllers (PLCs), the industrial computers that run plants, and refuses to declare a task finished until logged external checks - a specification review, compilation, and execution on a live PLC runtime - all pass, rather than trusting the model's own judgment.
It was evaluated on two tracks: 117 independent code-unit tasks from an existing benchmark, and 65 tasks built from ten real industrial plants where generated logic must integrate into and run inside an existing project.
On the standalone-unit track, SemaPLC achieved the highest strict verified pass rate on all seven models (72.6% average), 17.3 points higher than the same setup with the harness stripped out.
On the project-integration track, it had the best average compile success rate (89.4%); static, text-only checks showed only small gaps between methods, but when logic was actually deployed and its execution trace compared to a reference, SemaPLC scored 52.2 on average versus at most 31.4 for other methods.
Its advantage narrowed on the strongest model tested (GPT-5.5), and because the runtime tests use a fixed, pre-derived set of scenarios, performance under unseen operating conditions remains unmeasured.
Figure 1: Overview of the SemaPLC agent harness: a model-agnostic agent core, grounded in the task or project context, acts through a shared PLC MCP tool layer, and a verification gate decides completion.
Table 1: Function track: strict verified pass rate (%, denominator 117; inconclusive and empty generations count as failure). All methods are graded by the same held-out judge. bare is SemaPLC with the harness stripped from the same backbone (no skills and tools); full is the complete harness. DS = DeepSeek.
SemaPLC
Model
LLM4PLC
AutoPLC
Agents4PLC
bare
full
MiniMax-M2.7
22.2
49.6
53.8
39.3
69.2
MiniMax-M3
15.4
65.0
55.6
60.7
69.2
Qwen3.5-Plus
13.7
67.5
67.5
62.4
75.2
DS-V4-Flash
41.0
54.7
54.7
34.2
67.5
DS-V4-Pro
43.6
61.5
62.4
55.6
69.2
GLM-5.2
30.8
59.0
74.4
63.2
76.1
GPT-5.5
44.4
79.5
78.6
71.8
82.1
Mean
30.2
62.4
63.9
55.3
72.6
Worst
13.7
49.6
53.8
34.2
67.5
Table 2: Project-context track (0–100, best per column in bold): integrated-project compile rate, assertion-oracle static score, and live-runtime dynamic score, all over the full 65-task denominator. Worst/Best/Mean summarize each method over the seven models.
MiniMax
MiniMax
Qwen3.5
DeepSeek
DeepSeek
Method
M2.7
M3
Plus
V4-Flash
V4-Pro
GLM-5.2
GPT-5.5
Worst
Best
Mean
Integrated compilation
LLM4PLC
47.7
53.8
16.9
60.0
52.3
80.0
100.0
16.9
100.0
58.7
AutoPLC
69.2
95.4
58.4
58.5
95.4
95.4
98.5
58.4
98.5
81.5
Agents4PLC
47.7
69.2
40.0
75.4
78.5
89.2
98.5
40.0
98.5
71.2
SemaPLC
81.5
95.4
80.0
84.6
89.2
95.4
100.0
80.0
100.0
89.4
Static behavior
LLM4PLC
76.1
76.7
74.5
70.3
69.2
77.1
86.3
69.2
86.3
75.7
AutoPLC
68.9
73.5
68.9
76.2
66.5
75.5
88.8
66.5
88.8
74.0
Agents4PLC
63.4
73.5
77.6
71.4
64.5
62.8
88.6
62.8
88.6
71.7
SemaPLC
74.9
84.9
79.9
78.0
81.2
88.0
84.1
74.9
88.0
81.6
Dynamic behavior
LLM4PLC
3.0
26.1
6.6
18.4
13.9
34.6
54.5
3.0
54.5
22.4
AutoPLC
4.0
43.5
19.8
23.9
45.7
21.9
61.1
4.0
61.1
31.4
Agents4PLC
4.5
30.8
11.4
28.3
28.8
44.6
63.6
4.5
63.6
30.3
SemaPLC
31.3
52.1
43.1
54.1
57.4
61.9
65.4
31.3
65.4
52.2
Table 3: Project-track verification-layer ablation on DeepSeek-V4-Flash (65 tasks), adding checks cumulatively. Comp./Static/Dyn. use the Table 2 metrics; Tok./Reqs. are mean tokens and model requests per task.
Verification layers
Comp.
Static
Dyn.
Tok.
Reqs.
None (generate only)
64.6
71.5
23.1
34k
8.9
+ Spec
70.8
74.0
30.3
60k
14.6
+ Compile
83.1
77.8
43.7
74k
25.5
+ Runtime (full)
84.6
78.0
54.1
129k
47.8
Table 4: Outcome shares (%) of the 3,590 golden-reference scenario–port value checks (65 project-track tasks, DeepSeek-V4-Flash) under the cumulative arms of Table 3. The Correct row equals the Dyn. column of Table 3; subshares may not sum exactly to the totals due to rounding.
Outcome
None
+Spec
+Compile
+Runtime
Check failed
76.9
69.7
56.3
45.9
not built/run
33.4
24.5
19.7
3.3
port missing
41.3
35.7
20.0
18.2
wrong value (normal)
0.9
3.0
4.9
7.1
wrong value (breach)
1.2
6.5
11.7
17.3
Correct
23.1
30.3
43.7
54.1
Table 5: Function-track formal-verification coverage of the held-out pipeline, pooled over 1,293 delivered programs from SemaPLC runs. Conclusive means satisfied or violated.
Program group
Programs
Props.
Concl. %
Inconcl. %
No REAL/timer
916
4653
75.7
24.3
With REAL
345
1845
87.0
13.0
With timer (TON)
32
174
0.0
100.0
Table 6: Interaction cost per task against the strongest baseline: requests and end-to-end wall-clock seconds. Values are per-model per-task aggregates (means for requests, medians for times) averaged over the seven models, with across-model ranges in parentheses. Both methods call the same endpoints.
Method
Requests / task
Time / task (s)
Function track
Agents4PLC
6.3 (4.4 – 7.2)
454 (241 – 688)
SemaPLC
6.5 (5.5 – 7.6)
71 (41 – 156)
Project track
Agents4PLC
6.9 (6.8 – 7.0)
344 (47 – 917)
SemaPLC
34.1 (16.4 – 60.4)
347 (25 – 1380)
Table S1: PLC tools exposed over the Model Context Protocol.
Tool
Function
plc_check
Syntax/semantic check of a bare POU with the same compiler as the evaluation
plc_compile
Full ST compilation with structured, line-anchored diagnostics
plc_detectIO
Extract the located-I/O surface (addresses, types, directions)
plc_upload
Upload the compiled program to the runtime and return build logs
plc_buildAndRun
Compile, upload, and start in one call with per-stage results
plc_start
Start the loaded program
plc_stop
Stop the running program
plc_status
Runtime state query
plc_getLogs
Runtime logs with an error flag
plc_readVariables
Read live variable values over the debug protocol
plc_forceVariables
Force or release input values to simulate external signals
plc_trace
Sample variables over time (timers, state machines, counters)
plc_record
Fetch the per-scan recording of transitions for scan-precise sequences
plc_waitFor
Poll one variable until a comparison holds or times out
plc_verifyBehavior
Atomic force, expect, and release check of a downstream effect
plc_buildSimulation
Build an animated process-simulation scene for the running program
Table S2: Command-line entry points of the same tool server.
Command
Function
verify
Run a declarative verify plan (build, drive cases, assert, clean up) and emit a JSON verdict envelope
compile
Compile an ST file and print the structured result
buildAndRun
Compile, upload, and start: the full deploy loop from the shell
detectIO
Extract the located-I/O map from an ST file
genModbusConfig
Generate the Modbus slave configuration from the located I/O
readVariables
Read runtime variable values
force
Force or release runtime variables to simulate inputs
trace
Sample variables over time
waitFor
Poll a variable until the comparison holds or times out
genScene
Suggest a process-simulation scene from the located I/O
status
Quick runtime status check
serve
Start the MCP stdio server under a chosen tool profile
Table S3: The skill library, one skill per verification layer. Skill names are abbreviated.
Skill
Role
spec-review
Requirement-to-logic checklist run before delivery, covering named devices and signals, function-block instance calls, and published-variable drivers
fix-compile-error
Repair the earliest diagnostic only, recheck, at most two rounds, then report stuck
benchmark-verify
Runtime behavior check: build an injectable test copy, deploy it, drive self-derived scenarios, assert published outputs
Table S4: Information seen by the agent during generation versus information used for final scoring.
Information / result
Seen by method?
Scores result?
Requirement + context
Yes
No
Compiler diagnostics
Yes
Yes
Agent’s own audit feedback
Yes
No
Held-out function judge
No
Yes
Project assertion oracle
No
Yes
Golden reference / trace
No
Yes
Why it matters
PLCs control factories, power plants, and water treatment facilities, and code that merely compiles can still hide timer errors or wrong state transitions that only show up during actual operation - a serious safety concern. For engineers considering AI-generated control code, this shows that static checks alone can't tell reliable systems from unreliable ones; real execution testing is necessary.
Terms in this paper
PLC (Programmable Logic Controller) · An industrial computer that automates control of machinery and plant equipment
POU (Program Organization Unit) · A self-contained function or block that makes up part of a PLC program
Structured Text (ST) · A text-based programming language used for PLCs
MCP (Model Context Protocol) · A standardized way for AI models to call external tools
dynamic/runtime validation · Checking correctness by actually running the code and comparing its live behavior to a reference
Original abstract (English)
Programmable logic controllers (PLCs) run industrial plants, and large language models can already generate independent program organization units (POUs) for them. Whether such logic integrates into an existing PLC project and then runs correctly has been checked only in limited tests. We present SemaPLC, a project-grounded and verification-gated agent harness assembled from conventional tools but governed by a strict completion rule. Rather than stopping when the model judges its own output adequate, SemaPLC declares a task complete only when logged external checks confirm it. Those checks cover the specification, the compilation, and the behavior on a live runtime. On 117 independent-POU tasks matching existing benchmarks, it attains the highest strict verified pass rate on all seven models (72.6\% mean). On a project-context track of 65 tasks whose generated logic must compile and run inside a real project, it attains the highest mean on integrated compilation, static behavior, and dynamic behavior. Of the three layers, dynamic behavior is the most revealing. We measure it by deploying the generated and the reference logic to a live PLC runtime and comparing their executed traces. All methods fall within 10 static points of one another, whereas dynamic scores separate them sharply, from 22.4 to 31.4 for the baselines against 52.2 for SemaPLC. Overall, our verification-gated harness raises the mean at every layer and most sharply at runtime. Execution, not static scoring, is the faithful test of whether generated control logic actually works. SemaPLC is open-sourced at https://github.com/midea-ai/SemaPLC.
