AI 에이전트가 컴퓨터 작업을 '진짜로' 끝냈는지 판정하는 AI 채점관들이 사실은 잘 속는다는 것을 처음으로 체계적으로 밝혀낸 연구

arXiv:2607.286092026-07-29

OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

화면을 보고 클릭하고 타이핑하며 컴퓨터를 대신 조작하는 AI 에이전트(CUA)가 일을 제대로 끝냈는지는 결국 다른 AI(VLM, 비전-언어 모델)가 판정하는데, 이 판정관이 얼마나 믿을 만한지는 제대로 검증된 적이 없었다. 연구팀은 웹·모바일·우분투·윈도우 네 플랫폼에서 사람이 직접 검증한 1019개의 실제 작업 기록을 모아 OSReward라는 벤치마크를 만들고 27개의 VLM 판정관을 시험했는데, 최고 성능 모델조차 어려운 케이스에서 정확도가 70%대로 떨어졌고 대부분은 '실패한 작업을 성공으로 착각'하는 관대함 편향을 공유했다. 이에 연구팀은 10만 건 규모의 학습 데이터(OS-Shepherd-100K)를 만들어 저렴하면서도 상용 모델과 맞먹는 판정 성능을 내는 오픈소스 리워드 모델 OS-Shepherd(9B, 35B)를 학습시켜 공개했다.

METAL MEDIA 해설 도표

OSReward에서 OS-Shepherd까지의 흐름

증거 상태측정 결과가 보고됨

1. 실환경 구축웹·모바일·우분투·윈도우에 실제 사용자처럼 앱, 로그인 계정, 파일, 미끼 데이터를 채운 환경을 직접 구축
2. 에이전트 실행 및 사람 정답 라벨링4개 모델 계열 에이전트가 검증된 지시문을 수행, 3인 독립 평가 후 이견 시 상급 검토로 1019개 정답 세트(OSReward/Hard/Multi) 완성
3. 27개 VLM 판정관 시험정답 세트를 기준으로 프론티어부터 소형 오픈모델까지 27개를 시험, 관대함 편향과 어려운 사례에서의 붕괴 확인
4. OS-Shepherd-100K 코퍼스 구축32만여 건의 판정 사례를 정제해 10만 건 규모의 추론 근거 포함 학습 데이터 생성
5. OS-Shepherd 훈련 및 검증SFT 후 GRPO 강화학습으로 9B/35B 리워드 모델 학습, 상용 모델 대비 30~60배 저비용으로 근접한 정확도 확인

METAL MEDIA이 원문을 바탕으로 재구성한 해설 도표이며, 논문 저자의 원문 figure가 아닙니다.

무엇을 했나

기존 벤치마크의 재사용 기록물은 품질 문제와 부정확한 정답 표시가 뒤섞여 있어, 연구팀은 웹·모바일·우분투·윈도우 네 플랫폼에 실제 사용자 환경(로그인된 계정, 실제 파일, 미끼 데이터 등)을 직접 구축해 새 작업 기록을 처음부터 수집했다
Claude, Gemini, Kimi, Qwen 등 네 모델 계열의 에이전트에게 사람이 검증한 지시문을 수행시켜 성공과 실패가 실제로 섞인 1019개 기록을 얻었고, 각 기록은 3명의 독립 평가자가 라벨링하고 의견이 갈리면 상급 검토자 2인이 최종 판정하는 방식으로 약 800시간의 사람 작업을 들여 정답을 만들었다
이 정답 세트를 전체(OSReward), 특히 어려운 사례만 모은 OSReward-Hard(284개, 실패 비율 70%로 조정), 성공 사례의 효율성·의도 부합도까지 매긴 OSReward-Multi(440개)로 나눠 27개 VLM 판정관을 같은 조건에서 시험했다
최고 판정관(Claude-Opus-4-8)조차 전체 세트에서 89.7% 정확도였지만 OSReward-Hard에서는 69.7%로 추락했고, 평균 판정관은 52%까지 떨어졌으며, 전체 오류의 3분의 2가 '완료 안 된 작업을 성공으로 인정'하는 오류였다
이 관대함 편향을 개선하기 위해 32만여 건의 판정 데이터를 정제해 만든 10만 건 코퍼스(OS-Shepherd-100K)로 두 단계(SFT 후 강화학습)로 학습시킨 OS-Shepherd 모델은 상용 최고 모델 대비 30~60배 낮은 비용으로 비슷한 판정 성능을 보였다

Figure 1: Cost against binary accuracy on OSReward-Hard: reliable judges are expensive, and the OS-Shepherd models come nearest their accuracy at a fraction of the cost.

Table 1: Main-setting results for the reference judges and OS-Shepherd on OSReward and OSReward-Hard along with their access status, sorted by full-set accuracy.
Judge	Access	OSReward	OSReward-Hard
Acc	sRec	fRec	BalAcc	Acc	sRec	fRec	BalAcc
Claude-Opus-4-8	closed	89.7	91.1	88.9	90.0	69.7	69.8	69.7	69.7
GPT-5.5	closed	89.5	91.8	87.8	89.8	67.3	66.3	67.7	67.0
Claude-Opus-4-6	closed	89.5	92.7	87.7	90.2	67.3	72.1	65.2	68.6
Gemini-3.1-Pro	closed	87.9	90.2	86.2	88.2	61.6	61.6	61.6	61.6
Gemini-3.5-Flash	closed	87.8	95.7	81.8	88.8	59.5	81.4	50.0	65.7
Claude-Sonnet-4-6	closed	87.7	97.5	80.3	88.9	59.2	90.7	45.5	68.1
GPT-5	closed	87.4	86.8	87.9	87.4	58.1	43.0	64.6	53.8
GPT-5.4	closed	87.1	87.3	87.0	87.1	63.0	62.8	63.1	63.0
Gemini-3-Flash	closed	87.0	96.6	79.8	88.2	57.0	86.0	44.4	65.2
GPT-5-mini	closed	86.1	93.8	80.2	87.0	56.3	79.1	46.5	62.8
Kimi-K2.5	open weights	85.9	95.5	79.2	87.3	54.8	83.7	42.1	62.9
Qwen3.5-397B-A17B	open weights	85.8	95.2	78.6	86.9	58.5	91.9	43.9	67.9
GPT-5.4-mini	closed	85.2	82.5	87.2	84.9	58.1	48.2	62.4	55.3
Claude-Haiku-4-5	closed	84.5	80.9	87.2	84.0	59.5	47.7	64.6	56.2
GPT-5.2	closed	83.9	73.0	92.2	82.6	63.0	30.2	77.3	53.8
Gemini-2.5-Flash	closed	83.3	95.5	74.0	84.8	48.9	90.7	30.8	60.8
Doubao-2.0-Lite	closed	83.3	98.5	72.1	85.3	45.5	96.1	24.3	60.2
GPT-5-nano	closed	82.3	97.0	71.1	84.1	45.4	95.3	23.7	59.5
Intern-S1-Pro	open weights	82.3	92.3	74.7	83.5	43.7	70.9	31.8	51.4
Qwen3.5-35B-A3B	open weights	82.2	92.4	74.5	83.5	51.1	83.7	36.9	60.3
Qwen3.5-27B	open weights	82.0	97.4	70.5	84.0	44.2	92.9	23.2	58.0
GPT-4o	closed	81.0	96.8	69.0	82.9	39.4	90.7	17.2	53.9
Intern-S2-Preview	open weights	80.6	98.4	66.9	82.7	40.3	94.2	16.8	55.5
Qwen3.5-122B-A10B	open weights	79.6	96.8	66.4	81.6	39.4	89.5	17.7	53.6
Qwen3-VL-8B	open weights	77.1	99.8	59.9	79.8	36.2	100.0	8.2	54.1
Qwen3-VL-235B	open weights	74.0	99.1	54.9	77.0	31.4	97.7	2.5	50.1
Qwen3-VL-30B	open weights	69.4	99.8	46.3	73.0	31.1	98.8	1.5	50.2
OS-Shepherd-9B (ours)	open weights + data	86.1	86.6	86.0	86.3	60.2	66.3	57.6	61.9
OS-Shepherd-35B-A3B (ours)	open weights + data	85.6	85.0	86.2	85.6	62.7	68.6	60.1	64.3

Figure 2: From realistic environments to raw trajectories: annotators prepare the environments and write grounded instructions on them, agents from four model families execute the instructions.

Table 2: Strong judges on OSReward-Multi (%), sorted by AUC; best per column in bold.
Judge	Macro-recall	AUC
Align	Effic	Multi
GPT-5.5	58.7	68.2	63.5	66.7
Claude-Opus-4-8	52.9	68.7	60.8	65.6
Claude-Sonnet-4-6	53.2	62.6	57.9	61.9
Gemini-3.5-Flash	47.6	71.4	59.5	60.8
OS-Shepherd-35B-A3B (ours)	47.7	65.8	56.8	60.7
OS-Shepherd-9B (ours)	44.1	54.0	49.0	58.5
Gemini-3-Flash	50.6	61.5	56.0	55.8

Figure 3: The annotation pipeline. Each pre-filtered trajectory is labeled by three independent annotators; disagreements go to meta review, and the verified gold set is read as three views

Table 3: OS-Shepherd-100K judge-instance pool by platform.
Platform	Instances	Share
Web	119,469	37%
Windows	62,053	19%
macOS	45,028	14%
Ubuntu (GUI only)	34,355	11%
Ubuntu (GUI + CLI)	29,785	9%
Mobile	30,941	10%
Total	321,631	100%

Figure 4: OSReward at a glance: outcome composition of the full and Hard sets, platform mix, and trajectory length; failed runs are markedly longer.

Table 4: OS-Shepherd against its untuned base, on the full set and OSReward-Hard.
Model	OSReward	OSReward-Hard
Acc	sRec	fRec	BalAcc	Acc	sRec	fRec	BalAcc
Qwen3.5-9B (base)	76.7	98.9	59.9	79.4	39.4	97.7	14.1	55.9
OS-Shepherd-9B	86.1	86.6	86.0	86.3	60.2	66.3	57.6	61.9
Qwen3.5-35B-A3B (base)	82.2	92.4	74.5	83.5	51.1	83.7	36.9	60.3
OS-Shepherd-35B-A3B	85.6	85.0	86.2	85.6	62.7	68.6	60.1	64.3

Figure 5: Judge bias on OSReward (left) and OSReward-Hard (right). Judges below the diagonal skew lenient; most of the field sits there, and the skew widens on the hard set.

Table 5: Action spaces of the executing agents on web (left) and mobile (right). On web, the second block lists the browser primitives and the third the terminal action.
Action	Description
click [coord]	Clicks at the specified screen location.
double_click [coord]	Double-clicks at the specified screen location.
hover [coord]	Moves the pointer to the specified screen location.
scroll [up/down]	Scrolls the screen in the specified direction.
drag [coord] [coord]	Drags from the first coordinate to the second.
type [text]	Types text at the current cursor location.
fill [coord] [text]	Clicks at a location, clears its content, and types text.
clear [coord]	Clicks at a location and clears the current text input.
hotkey [keys]	Presses the specified key or key combination.
wait [seconds]	Waits for the page to load.
goto [url]	Navigates directly to a URL.
go_back	Navigates to the previous page in browser history.
go_forward	Navigates to the next page in browser history.
select_option [coord] [text]	Selects text from the dropdown at a screen location.
set_checked [coord] [bool]	Sets the control state at a screen location.
stop [answer]	Terminates the episode and returns the final answer.

Figure 6: Per-judge error composition, seven representative judges: over-accepting an incomplete task (warm colors) dominates every family.

Table 6: Action space of the executing agents on Windows, with each action’s parameter format.
Action	Parameter specification
computer.mouse.move_abs	Format: [x,y] Details: Move the mouse to a normalized screen position; x, y are floats.
computer.mouse.single_click	Format: [] Details: Single-click at the current mouse position.
computer.mouse.double_click	Format: [] Details: Double-click at the current mouse position.
computer.mouse.right_click	Format: [] Details: Right-click at the current mouse position.
computer.mouse.scroll	Format: [direction] Details: Scroll the screen up or down; direction is a string.
computer.mouse.drag	Format: [x1,y1,x2,y2] Details: Drag from the current mouse position to the target normalized position; coordinates are floats.
computer.keyboard.write	Format: [text] Details: Type the given text.
computer.keyboard.press	Format: [key] Details: Press a keyboard key such as Enter or Delete.
computer.os.open_program	Format: [program_name] Details: Open the specified application.
computer.window_manager.switch_to_application	Format: [window_name] Details: Switch to the specified open window or application.
computer.wait	Format: [time] Details: Wait for the given number of milliseconds (time is an integer).
COMMAND	Format: [] Details: Output and execute a Python code block for the current step.
ANSWER	Format: [answer] Details: Return the specific answer text for the given prompt.
DONE	Format: [] Details: The task is finished; end the episode.
FAIL	Format: [] Details: The task cannot be completed; end the episode.

Figure 7: Mean per-judge binary accuracy on OSReward-Hard, by platform and by failure type. Failure types are multi-label, so their counts sum to more than the number of failed trajectories.

Table 7: Application coverage of the collection infrastructure on Ubuntu, Windows, and Android, grouped by function; applications marked with ∗ require a signed-in account. The web platform runs on live websites rather than installed applications (Section A.1).
Platform	Group	Applications
Ubuntu	Web & communication	Chrome, Thunderbird, Zoom∗
Development	VS Code, PyCharm, GitKraken, DBeaver, Wireshark, Meld, terminal
Documents	LibreOffice Writer / Calc / Impress, TeXstudio, PDF Arranger, Zotero, Calendar
Graphics & design	GIMP, Blender, Inkscape, Krita, Darktable, LibreCAD, KiCad, draw.io
Media	VLC, Audacity, Mixxx, HandBrake, Shotcut, OBS Studio, MuseScore, Spotify∗
Scientific	Scilab, KAlgebra, GRASS GIS, Google Earth Pro, ChimeraX, Celestia
Personal	HomeBank
Windows	Web & communication	Chrome, Microsoft Edge, Thunderbird, Feishu∗, Discord∗, Zoom∗, Tencent Meeting∗
Development	VS Code, PyCharm, DBeaver
Documents	Notepad, PDF Arranger, Zotero
Graphics & design	Blender, Krita, draw.io
Media	VLC, Shotcut, HandBrake, Spotify
Utilities	File Explorer, Calculator
Personal	Steam∗
Android	Web & communication	Browser, Firefox, Gmail∗, SMS, Contacts
Documents	Markor, Google Keep∗, Calendar
Graphics & design	Draw
Media	Camera, Gallery, Google Photos∗, Audio Recorder, VLC, Retro Music
Maps & navigation	OsmAnd, Google Maps
Personal	Expense, Recipe, Yahoo Finance
Utilities	Files, Clock, Calculator, and system tasks

Figure 8: Overview of input ablations: Δ binary accuracy per (setting × model) vs. the main setting.

Table 8: Action space of the executing agents on Ubuntu, with each action’s parameter format.
Action	Parameter specification
click	Format: [desc,num_clicks,button,hold_keys] Details: Target element description; clicks number; button to click; keys to hold.
type	Format: [desc,text,overwrite,enter,terminal] Details: Target element description; text content; overwrite flag (bool); press enter after typing (bool); terminal flag (bool).
scroll	Format: [desc,clicks,shift] Details: Target element description; clicks (+up/−down); shift for horizontal scroll (bool).
drag_and_drop	Format: [start_desc,end_desc,hold_keys] Details: Descriptions for start/end locations; keys to hold during drag.
hotkey	Format: [keys] Details: List of keys to press in combination (e.g., [‘ctrl’, ‘c’]).
hold_and_press	Format: [hold_keys,press_keys] Details: Keys to hold down while pressing a sequence of other keys.
open	Format: [app_or_filename] Details: Name of the application or file to open.
call_code_agent	Format: [task] Details: A self-contained goal executable via code (e.g., data analysis, file processing).
wait	Format: [time] Details: Time to wait in seconds.
done	Format: [] Details: Signals successful completion of the entire task.
fail	Format: [] Details: Signals that the task is impossible to complete.

Figure 9: The OS-Shepherd-100K pipeline: self-collected and open-source trajectories are ensemble-judged and distilled into the training set. Band widths ∝ trajectory counts.

Table 9: All evaluated models: the 27 reference judges (top, by full-set accuracy) and our reward models. The last column lists the extra thinking or reasoning-effort levels beyond the main setting; access classes are in Table 1.
Judge	API identifier	Thinking levels
Claude-Opus-4-8 (Anthropic 2026b)	claude-opus-4-8	—
GPT-5.5 (OpenAI 2026b)	gpt-5.5	medium/high/xhigh
Claude-Opus-4-6 (Anthropic 2026a)	claude-opus-4-6	xhigh/max
Gemini-3.1-Pro (Gemini Team 2025)	gemini-3.1-pro-preview	—
Gemini-3.5-Flash (Gemini Team 2025)	gemini-3.5-flash	—
Claude-Sonnet-4-6 (Anthropic 2026c)	claude-sonnet-4-6	xhigh/max
GPT-5 (OpenAI 2025b)	gpt-5	—
GPT-5.4 (OpenAI 2026a)	gpt-5.4	—
Gemini-3-Flash (Gemini Team 2025)	gemini-3-flash-preview	—
GPT-5-mini (OpenAI 2025b)	gpt-5-mini	—
Kimi-K2.5 (Kimi Team et al. 2026)	kimi-k2.5	—
Qwen3.5-397B-A17B (Qwen Team 2026)	qwen3.5-397b-a17b	two settings
GPT-5.4-mini (OpenAI 2026a)	gpt-5.4-mini	—
Claude-Haiku-4-5 (Anthropic 2025)	claude-haiku-4-5-20251001	—
GPT-5.2 (OpenAI 2025b)	gpt-5.2	—
Gemini-2.5-Flash (Comanici et al. 2025)	gemini-2.5-flash	—
Doubao-2.0-Lite (ByteDance Seed Team 2026)	doubao-seed-2-0-lite-260428	—
GPT-5-nano (OpenAI 2025b)	gpt-5-nano	—
Intern-S1-Pro (Zou et al. 2026)	intern-s1-pro	—
Qwen3.5-35B-A3B (Qwen Team 2026)	qwen3.5-35b-a3b	—
Qwen3.5-27B (Qwen Team 2026)	qwen3.5-27b	—
GPT-4o (Hurst et al. 2024)	gpt-4o	—
Intern-S2-Preview (Zou et al. 2026)	intern-s2-preview	—
Qwen3.5-122B-A10B (Qwen Team 2026)	qwen3.5-122b-a10b	—
Qwen3-VL-8B (Bai et al. 2025)	qwen3-vl-8b-instruct	two settings
Qwen3-VL-235B (Bai et al. 2025)	qwen3-vl-235b-a22b-instruct	—
Qwen3-VL-30B (Bai et al. 2025)	qwen3-vl-30b-a3b-instruct	—
OS-Shepherd-9B (ours)	os-shepherd-9b	—
OS-Shepherd-35B-A3B (ours)	os-shepherd-35b-a3b	—

Figure 10: Judges on three existing CUA benchmarks against each benchmark’s human-written verifier (matched subsets): accuracy and failure recall; means are computed before rounding.

Table 10: OSReward beside existing CUA reward works, on data provenance and released artifacts rather than head-to-head scores (their input formats and platform scopes preclude a shared protocol). OSReward is the only one built end-to-end from freshly collected, human-gold trajectories and the only one whose gold goes beyond a binary verdict. Platforms W/M/D = web/mobile/desktop; Instr. / Traj. / Gold flag fresh instructions, fresh trajectories, and human-labeled gold; Corpus / Model give any released training corpus and reward model (✓ yes, ✗ no, ∼ partial, – n/a).
			Reward benchmark	Reward model
Dataset	Platforms	Action	Instr.	Traj.	Gold	Labels	Corpus	Model
OSReward (ours)	W, M, D	GUI+CLI	✓	✓	✓	Binary + fine-grained	✓ 100K	✓ 9B/35B
Web-Shepherd Chae et al. 2025	W	GUI	∼	∼	∼	Checklist	✓ 40K	✓ 3B/8B
GUI-Shepherd Chen et al. 2025a	M	GUI	–	–	–	–	✓ 52K	✓ 7B
CUARewardBench Lin et al. 2025	D	GUI	✗	✗	✓	Binary	–	–
OS-Themis Li et al. 2026	W, M, D	GUI	✗	✓	✗	Binary	–	–

Figure 11: The twenty most frequent task types among the roughly 32K web instructions sampled for collection, together about 65% of the pool. Fact-finding lookups dominate, followed by article, academic-paper, and shopping tasks.

Table 11: The OS-Shepherd-100K judge-instance pool by source (321,631 instances over eight sources). success is the share of agent-successful verdicts per source; the web pool is the most failure-rich. Nothing is drawn from any existing benchmark’s test set (Sections A.1 and 7).
	Source	Platform	Instances	success
Self-collected	Web	Web	117,251	45%
Ubuntu (GUI+CLI)	Ubuntu	29,785	72%
Scientific (Sun et al. 2026b)	Ubuntu	14,339	59%
Windows	Windows	3,599	50%
OS-Genesis (re-generated; Sun et al. 2025a)	Web	2,218	73%
Reused	OpenCUA (Wang et al. 2025)	Windows / macOS	103,482	69%
OpenMobile (Cheng et al. 2026)	Mobile	30,941	62%
OpenCUA (Wang et al. 2025)	Ubuntu	18,916	78%
ScaleCUA (Liu et al. 2026)	Ubuntu	1,100	64%

Figure 12: Failure-type profile over OSReward’s fail trajectories (multi-label shares; the catch-all others tag is excluded). A single run can carry several tags.

Table 12: Screenshot-setting mix of the retained training samples.
Screenshot setting	Share
Last-5 frames	45.1%
First-1 + last-2	26.0%
Last-3 frames	18.9%
Last-10 frames	8.9%
Last-6/7/8 frames	1.2%

Table 13: OS-Shepherd training configuration for both sizes. SFT is shared (same corpus and schedule); the two RL runs share the mined set and differ only in the base checkpoint.
	SFT	RL
	(both sizes)	9B	35B-A3B
Base model	Qwen3.5-9B / Qwen3.5-35B-A3B	9B SFT ckpt	35B SFT ckpt
Samples	96.6K	3.1K (shared)
Rollouts / sample	—	8 (at T=1.0, top-p 1.0)
Batch size	—	16
Learning rate	—	1e−6
KL to SFT ref.	—	0.001 (low-variance, as loss)
Max prompt / resp.	—	24,576 / 512 tokens
Steps	1 epoch	∼150 (≈1 pass)
Framework	verl + SGLang rollout back-end
Hardware	32× NVIDIA H200 (4 nodes × 8)

Table 14: OS-Shepherd-9B beside its full-set accuracy tier and two frontier judges. Cost is list price to judge the full set; full/hard are binary accuracy (%).
Judge	Weights	Cost ($)	Full	Hard
Claude-Opus-4-8	closed	86.04	89.7	69.7
GPT-5.5	closed	45.44	89.5	67.3
Kimi-K2.5	open	20.37	85.9	54.8
Qwen3.5-397B-A17B	open	7.96	85.8	58.5
GPT-5.4-mini	closed	6.20	85.2	58.1
GPT-5-mini	closed	2.17	86.1	56.3
Gemini-3-Flash	closed	2.02	87.0	57.0
OS-Shepherd-9B (ours)	open	1.36	86.1	60.2
Qwen3.5-9B	open	1.36	76.7	39.4

Table 15: Thinking and reasoning effort. Each left-hand row contrasts two settings of one model, so Δ is within-model; the Qwen3-VL-8B thinking arm rejects ∼6%, making its Δ intersection-paired. Right: the GPT-5.5 reasoning-effort sweep.
Model	Setting	Acc	Setting	Acc	Δ
Qwen3-VL-8B	no thinking	77.1	thinking	81.7	+2.83
Qwen3.5-397B-A17B	no thinking	85.8	thinking	86.7	+0.89
Claude-Sonnet-4-6	xhigh	87.7	max	88.5	+0.59
Claude-Opus-4-6	xhigh	89.5	max	90.0	+0.39

실제로 확인된 결과

OSReward 전체 세트에서 최고 판정관 Claude-Opus-4-8이 89.7% 정확도를 기록했으나, 어려운 사례만 모은 OSReward-Hard에서는 모든 판정관의 정확도가 20~43퍼센트포인트 떨어졌고 최고 모델도 69.7%에 그쳤다
전체 오류의 약 3분의 2가 '완료되지 않은 작업을 성공으로 인정'하는 오류였고, 이 편향은 시험한 모든 판정관 계열에서 공통으로 나타났다(개별 모델 기준 최소 48% 이상)
OS-Shepherd-9B/35B는 동일한 판정 조건에서 상용 프론티어 모델과 비슷한 정확도를 30~60배 낮은 비용으로 달성했으며, 강화학습 단계에서 검증 정확도가 약 70%에서 약 77%로 올랐다
외부 벤치마크(OSWorld)에서 판정 오류의 88%가 허위 긍정이었고, 16단계를 넘는 긴 작업 기록일수록 정확도가 0.76에서 0.57로 떨어지고 허위 긍정률은 0.20에서 0.37로 올랐다

어디에 쓸 수 있나

AI 에이전트 훈련용 강화학습에서 저비용 리워드 신호로 OS-Shepherd 모델을 활용해 대량의 작업 기록을 채점하는 용도
기존 CUA 벤치마크나 데이터 수집 파이프라인에서 사람이 일일이 검증하기 힘든 대량의 작업 성공/실패 여부를 1차로 걸러내는 용도
새로운 VLM을 CUA 판정관으로 채택하기 전에 OSReward-Hard 같은 어려운 사례 세트로 관대함 편향을 사전 점검하는 용도

한계와 남은 검증

이 벤치마크는 웹·모바일·우분투·윈도우만 다루며 macOS는 다루지 않는다(학습 코퍼스의 macOS 데이터는 별도 공개 데이터에서 가져온 것)
판정 프로토콜은 마지막 5개 화면 캡처를 기본으로 사용하는 설정으로 고정되어 있어, 다른 입력 방식(도구 사용, 단계별 감독 등)에서의 신뢰도는 별도로 검증되지 않았다
논문은 '완료 확인을 명시적으로 요구하는 프롬프트'가 관대함 편향 완화에 더 효과적일 수 있다고 제안했지만 이는 향후 연구로 남겨두었다
OS-Shepherd의 강화학습 데이터는 허위 성공이 주로 데스크톱에 집중돼 있어 모바일과 웹 비중이 상대적으로 적게 반영됐다

왜 중요한가

AI 에이전트를 훈련시키거나 평가하려면 결국 다른 AI가 '이 작업이 성공했는가'를 판정해줘야 하는데, 그 판정관이 얼마나 믿을 만한지 몰랐다면 훈련 데이터와 평가 결과 자체가 왜곡될 수 있다. 이 연구는 그 판정관의 신뢰도를 처음으로 표준화된 방식으로 측정하고, 저비용으로도 쓸 만한 대안을 오픈소스로 내놓았다는 점에서 CUA 개발과 평가 전반에 실질적인 영향을 준다.

이 논문의 용어

CUA(Computer-Using Agent) · 화면을 보고 클릭·타이핑 등으로 컴퓨터를 직접 조작하는 AI 에이전트
VLM(비전-언어 모델) · 이미지(화면 캡처)와 텍스트를 함께 이해하는 AI 모델
false success(허위 성공) · 에이전트가 작업을 끝냈다고 주장하지만 실제로는 목표를 이루지 못한 경우
리워드 모델 · 다른 AI의 작업 결과를 평가해 강화학습 훈련이나 데이터 선별에 쓰이는 점수를 매기는 모델
GRPO · OS-Shepherd 강화학습 단계에서 사용된 정책 최적화 알고리즘

저자 · Qiushi Sun

arXiv에서 원문 보기

METAL MEDIA 최신 기사

그림 출처: Qiushi Sun et al., arXiv:2607.28609, CC BY 4.0

AI 에이전트가 컴퓨터 작업을 '진짜로' 끝냈는지 판정하는 AI 채점관들이 사실은 잘 속는다는 것을 처음으로 체계적으로 밝혀낸 연구

OSReward에서 OS-Shepherd까지의 흐름

무엇을 했나

실제로 확인된 결과

어디에 쓸 수 있나

한계와 남은 검증

왜 중요한가

이 논문의 용어

최신 논문

METAL MEDIA 최신 기사