Figure 1: Overview of the SemaPLC agent harness: a model-agnostic agent core, grounded in the task or project context, acts through a shared PLC MCP tool layer, and a verification gate decides completion.
Table 1: Function track: strict verified pass rate (%, denominator 117; inconclusive and empty generations count as failure). All methods are graded by the same held-out judge. bare is SemaPLC with the harness stripped from the same backbone (no skills and tools); full is the complete harness. DS = DeepSeek.
SemaPLC
Model
LLM4PLC
AutoPLC
Agents4PLC
bare
full
MiniMax-M2.7
22.2
49.6
53.8
39.3
69.2
MiniMax-M3
15.4
65.0
55.6
60.7
69.2
Qwen3.5-Plus
13.7
67.5
67.5
62.4
75.2
DS-V4-Flash
41.0
54.7
54.7
34.2
67.5
DS-V4-Pro
43.6
61.5
62.4
55.6
69.2
GLM-5.2
30.8
59.0
74.4
63.2
76.1
GPT-5.5
44.4
79.5
78.6
71.8
82.1
Mean
30.2
62.4
63.9
55.3
72.6
Worst
13.7
49.6
53.8
34.2
67.5
Table 2: Project-context track (0–100, best per column in bold): integrated-project compile rate, assertion-oracle static score, and live-runtime dynamic score, all over the full 65-task denominator. Worst/Best/Mean summarize each method over the seven models.
MiniMax
MiniMax
Qwen3.5
DeepSeek
DeepSeek
Method
M2.7
M3
Plus
V4-Flash
V4-Pro
GLM-5.2
GPT-5.5
Worst
Best
Mean
Integrated compilation
LLM4PLC
47.7
53.8
16.9
60.0
52.3
80.0
100.0
16.9
100.0
58.7
AutoPLC
69.2
95.4
58.4
58.5
95.4
95.4
98.5
58.4
98.5
81.5
Agents4PLC
47.7
69.2
40.0
75.4
78.5
89.2
98.5
40.0
98.5
71.2
SemaPLC
81.5
95.4
80.0
84.6
89.2
95.4
100.0
80.0
100.0
89.4
Static behavior
LLM4PLC
76.1
76.7
74.5
70.3
69.2
77.1
86.3
69.2
86.3
75.7
AutoPLC
68.9
73.5
68.9
76.2
66.5
75.5
88.8
66.5
88.8
74.0
Agents4PLC
63.4
73.5
77.6
71.4
64.5
62.8
88.6
62.8
88.6
71.7
SemaPLC
74.9
84.9
79.9
78.0
81.2
88.0
84.1
74.9
88.0
81.6
Dynamic behavior
LLM4PLC
3.0
26.1
6.6
18.4
13.9
34.6
54.5
3.0
54.5
22.4
AutoPLC
4.0
43.5
19.8
23.9
45.7
21.9
61.1
4.0
61.1
31.4
Agents4PLC
4.5
30.8
11.4
28.3
28.8
44.6
63.6
4.5
63.6
30.3
SemaPLC
31.3
52.1
43.1
54.1
57.4
61.9
65.4
31.3
65.4
52.2
Table 3: Project-track verification-layer ablation on DeepSeek-V4-Flash (65 tasks), adding checks cumulatively. Comp./Static/Dyn. use the Table 2 metrics; Tok./Reqs. are mean tokens and model requests per task.
Verification layers
Comp.
Static
Dyn.
Tok.
Reqs.
None (generate only)
64.6
71.5
23.1
34k
8.9
+ Spec
70.8
74.0
30.3
60k
14.6
+ Compile
83.1
77.8
43.7
74k
25.5
+ Runtime (full)
84.6
78.0
54.1
129k
47.8
Table 4: Outcome shares (%) of the 3,590 golden-reference scenario–port value checks (65 project-track tasks, DeepSeek-V4-Flash) under the cumulative arms of Table 3. The Correct row equals the Dyn. column of Table 3; subshares may not sum exactly to the totals due to rounding.
Outcome
None
+Spec
+Compile
+Runtime
Check failed
76.9
69.7
56.3
45.9
not built/run
33.4
24.5
19.7
3.3
port missing
41.3
35.7
20.0
18.2
wrong value (normal)
0.9
3.0
4.9
7.1
wrong value (breach)
1.2
6.5
11.7
17.3
Correct
23.1
30.3
43.7
54.1
Table 5: Function-track formal-verification coverage of the held-out pipeline, pooled over 1,293 delivered programs from SemaPLC runs. Conclusive means satisfied or violated.
Program group
Programs
Props.
Concl. %
Inconcl. %
No REAL/timer
916
4653
75.7
24.3
With REAL
345
1845
87.0
13.0
With timer (TON)
32
174
0.0
100.0
Table 6: Interaction cost per task against the strongest baseline: requests and end-to-end wall-clock seconds. Values are per-model per-task aggregates (means for requests, medians for times) averaged over the seven models, with across-model ranges in parentheses. Both methods call the same endpoints.
Method
Requests / task
Time / task (s)
Function track
Agents4PLC
6.3 (4.4 – 7.2)
454 (241 – 688)
SemaPLC
6.5 (5.5 – 7.6)
71 (41 – 156)
Project track
Agents4PLC
6.9 (6.8 – 7.0)
344 (47 – 917)
SemaPLC
34.1 (16.4 – 60.4)
347 (25 – 1380)
Table S1: PLC tools exposed over the Model Context Protocol.
Tool
Function
plc_check
Syntax/semantic check of a bare POU with the same compiler as the evaluation
plc_compile
Full ST compilation with structured, line-anchored diagnostics
plc_detectIO
Extract the located-I/O surface (addresses, types, directions)
plc_upload
Upload the compiled program to the runtime and return build logs
plc_buildAndRun
Compile, upload, and start in one call with per-stage results
plc_start
Start the loaded program
plc_stop
Stop the running program
plc_status
Runtime state query
plc_getLogs
Runtime logs with an error flag
plc_readVariables
Read live variable values over the debug protocol
plc_forceVariables
Force or release input values to simulate external signals
plc_trace
Sample variables over time (timers, state machines, counters)
plc_record
Fetch the per-scan recording of transitions for scan-precise sequences
plc_waitFor
Poll one variable until a comparison holds or times out
plc_verifyBehavior
Atomic force, expect, and release check of a downstream effect
plc_buildSimulation
Build an animated process-simulation scene for the running program
Table S2: Command-line entry points of the same tool server.
Command
Function
verify
Run a declarative verify plan (build, drive cases, assert, clean up) and emit a JSON verdict envelope
compile
Compile an ST file and print the structured result
buildAndRun
Compile, upload, and start: the full deploy loop from the shell
detectIO
Extract the located-I/O map from an ST file
genModbusConfig
Generate the Modbus slave configuration from the located I/O
readVariables
Read runtime variable values
force
Force or release runtime variables to simulate inputs
trace
Sample variables over time
waitFor
Poll a variable until the comparison holds or times out
genScene
Suggest a process-simulation scene from the located I/O
status
Quick runtime status check
serve
Start the MCP stdio server under a chosen tool profile
Table S3: The skill library, one skill per verification layer. Skill names are abbreviated.
Skill
Role
spec-review
Requirement-to-logic checklist run before delivery, covering named devices and signals, function-block instance calls, and published-variable drivers
fix-compile-error
Repair the earliest diagnostic only, recheck, at most two rounds, then report stuck
benchmark-verify
Runtime behavior check: build an injectable test copy, deploy it, drive self-derived scenarios, assert published outputs
Table S4: Information seen by the agent during generation versus information used for final scoring.
Programmable logic controllers (PLCs) run industrial plants, and large language models can already generate independent program organization units (POUs) for them. Whether such logic integrates into an existing PLC project and then runs correctly has been checked only in limited tests. We present SemaPLC, a project-grounded and verification-gated agent harness assembled from conventional tools but governed by a strict completion rule. Rather than stopping when the model judges its own output adequate, SemaPLC declares a task complete only when logged external checks confirm it. Those checks cover the specification, the compilation, and the behavior on a live runtime. On 117 independent-POU tasks matching existing benchmarks, it attains the highest strict verified pass rate on all seven models (72.6\% mean). On a project-context track of 65 tasks whose generated logic must compile and run inside a real project, it attains the highest mean on integrated compilation, static behavior, and dynamic behavior. Of the three layers, dynamic behavior is the most revealing. We measure it by deploying the generated and the reference logic to a live PLC runtime and comparing their executed traces. All methods fall within 10 static points of one another, whereas dynamic scores separate them sharply, from 22.4 to 31.4 for the baselines against 52.2 for SemaPLC. Overall, our verification-gated harness raises the mean at every layer and most sharply at runtime. Execution, not static scoring, is the faithful test of whether generated control logic actually works. SemaPLC is open-sourced at https://github.com/midea-ai/SemaPLC.
