VibeLifeBench: Can Your Life Agent Be Proactive and Persistent in a Living World?

arXiv:2608.108752026-08-12

让AI助手连续数周甚至数月处理旅行、财务、家务等生活事务时,即便是目前最强的模型也只能拿到约33分(满分100)

VibeLifeBench考察AI智能体能否在一个自行运转、持续数周乃至数月的模拟世界里主动察觉变化,自行判断何时行动、何时提问、何时保持沉默,并从始至终保持计划一致,而不是像以往那样只应对一次性的静态请求。该基准包含22个模拟服务、200个跨十大生活领域的任务,以及12261条细粒度评分检查项。表现最强的Claude Opus 5平均得分(avg@3)也只有32.5,其余六个模型均落在21到33分的狭窄区间,说明当前AI助手距离真正胜任日常生活协助还很远。

METAL MEDIA 解读图

VibeLifeBench任务结构:置身于自带时钟世界中的智能体

证据状态已报告实测结果

初始世界状态与人物设定22个模拟服务被预先设置好数据,智能体被赋予一个需要服务的人物角色,以及隐性约束和授权边界。
24个阶段的时间线,四类事件用户消息、世界观察和通知会开启智能体的一次回应,而变异事件在毫无提示的情况下悄悄改变世界状态,只有主动重新检查才能发现。
持续状态维护智能体需要把承诺记录在邮件、日历和笔记中,并在整个任务期间始终坚守护照有效期、预算上限、防范钓鱼邮件等隐藏约束。
三层评分体系分阶段检查、跨阶段检查、最终检查共12261条加权检查项,分别评判即时应对、全程约束坚守和最终留下的成果。
结果:七个模型全部得分偏低最强模型平均得分32.5,其余模型集中在21到33分之间,且所有模型在任务后期通过率都持续下降。

这是 METAL MEDIA 制作的解读图，并非论文作者提供的原图。

他们做了什么

现有智能体基准大多使用简短、一次性完成的请求,且环境在没有智能体行动的情况下永远不会改变,因此无法真正衡量助手在长期生活协助中的主动性表现,这正是本研究的出发点。
VibeLifeBench围绕出行、搬家、财务处理等十大生活领域构建了200个任务,每个任务都嵌入在一个由22个模拟服务(邮件、日历、银行、机票等)组成、按自身时钟持续运转的世界中。
事件分为四类:用户消息、世界观察、通知,以及在没有任何提示的情况下悄然改变世界状态的变异事件(mutation),变异事件约占全部事件的19.9%,只有主动重新查看世界的智能体才能发现它们。
每个任务都埋入了未明说的约束和安全红线,例如护照即将过期、胰岛素需要海关申报、钓鱼邮件等,并通过12261条加权检查项(每个任务平均58条)对智能体实际留下的痕迹进行评判。
研究评测了七个前沿模型(Claude Opus 5、GPT-5.5、Gemini 3.5 Flash、Claude Opus 4.8、GLM-5.2、Kimi-K2.6、DeepSeek-V4-Pro),结果全部得分偏低,且所有模型在任务时间线后三分之一阶段的通过率都比前三分之一低10到15分。

Figure 1: Overview of VibeLifeBench. The top shows a complete living-world task end to end: a timeline of about 30 simulated days and 24 stages, driven by four event kinds, some of which fire silently with no notification, so that only an agent that re-inspects the world on its own notices them in time; implicit constraints (passport validity, insulin customs declaration, a phishing email) must be proactively identified and upheld, and the agent must maintain durable state through to the end. The bottom shows the composition of the four event kinds, the classification of tasks into ten life domains, and the service backends that support the tasks.

Table 1: Comparison of VibeLifeBench with representative agent benchmarks. Proactive asks whether a task requires the agent to decide on its own, without prompting, when to act; living world asks whether the environment evolves on its own, independently of the agent; long-horizon asks whether a task is a multi-stage, long-cycle process with dependencies. ●, ◐, and ○denote satisfied, partially satisfied, and not satisfied.
Benchmark	Domain	Proactive	Living world	Long-horizon
SWE-Milestone [4]	Coding	○	○	●
Terminal-Bench [17]	Coding & terminal	○	○	◐
APEX-Agents [20]	Office & professional	○	○	◐
JobBench [11]	Office & occupational work	○	○	○
Workspace-Bench [19]	Office & knowledge work	○	○	○
UltraHorizon [14]	Synthetic exploration	◐	○	●
ClawBench [22]	Web	○	◐	○
UniClawBench [3]	Computer use	◐	◐	○
Claw-Eval [21]	General tool-use & dialogue	◐	◐	○
WildClawBench [5]	Office & computer use	○	◐	◐
CostBench [12]	Tool-use planning (travel)	◐	◐	○
ClawMark [16]	Office & knowledge work	◐	●	●
ClawArena [9]	Office & knowledge work	◐	●	●
VibeLifeBench (ours)	Life (ten domains)	●	●	●

Figure 2: Number of tasks that use each service across the suite (all 22 services are exercised). Usage is long-tailed.

Table 2: The four event kinds. The first three open an agent turn; a mutation is applied directly to the world state and produces no turn.
Event kind	Triggers a turn?	Meaning
User message	Yes	An utterance from the user (or a companion in the scenario), passed directly into the agent’s turn.
World observation	Yes	An external service reporting a world state (flight options, a visa rule, a market quote), entering the turn as an observation.
Notification	Yes	A system or channel push (a scheduled reminder, an operator alert), likewise surfaced to the agent.
Mutation	No	A background change to the world state (a flight quietly marked delayed, a phishing email placed in the inbox, a road-closure record inserted). It does not interrupt the agent; the world simply becomes different.

Figure 3: Per-domain distribution of horizon, number of events, number of services, and number of checks, as box plots.

Table 3: The evidence dimensions the scoring criteria cover.
Evidence dimension	What the check verifies
Tool call	Whether the agent called the right tool with the right arguments.
Backend end state	The final state of the backend services, such as orders, calendar, and ledger balances.
Persistent artifact	Text artifacts such as workspace files, notes, and calendar events.
Reply consistency	Whether the reply text is consistent with tool results and the authorization boundary.
Cross-stage consistency	Consistency across stages by combining several artifacts, such as a running ledger total and a red line that is never reversed.

Figure 4: Per-run input tokens, output tokens, tool calls, and turns.

Table 4: Per-domain composition of VibeLifeBench, reported as within-domain medians. Days is the simulated horizon (the maximum minus the minimum event timestamp, after removing a few sentinel end timestamps); events counts all four event kinds; services is the number of distinct mock services a task recruits.
Domain	Tasks	Med. days	Med. stages	Med. events	Med. services	Med. checks
travel	20	28	24	37	8	50
finance	20	20	24	36	6	94
litigation	20	33	25	32	5	52
renovation	20	29	24	40	8	68
career	20	48	24	44	7	43
fitness	20	34	28	30	5	50
exam preparation	20	40	24	33	6	52
rental	20	33	26	33	8	58
shopping	20	29	24	40	8	68
team building	20	24	25	31	8	59
Overall	200	29	24	36	7	58

Figure 5: Per-stage check pass rate as a function of the normalized position of the check along the task timeline, for the seven models.

Table 5: Main experimental performance and per-run token and interaction cost for the seven models. The within-task σ is the standard deviation of a task’s scores across its runs, averaged across tasks. Context read is the total context the model reads per run, in millions of tokens; output counts visible generation, and includes reasoning tokens only for the models that report them separately (GLM-5.2, Kimi-K2.6, DeepSeek-V4-Pro).
	Performance	Token & interaction cost
Model	avg@3	max@3	min@3	σ	Context (M)	Output	Tool calls	Turns
Claude Opus 5	32.5	41.2	23.8	9.8	30.2	325,198	316	210
GPT-5.5	30.1	38.8	21.5	10.0	17.6	78,631	332	146
Gemini 3.5 Flash	27.5	35.6	20.1	8.3	41.2	213,757	243	227
Claude Opus 4.8	27.5	34.3	20.3	7.5	28.8	220,795	228	111
GLM-5.2	25.4	29.9	20.9	4.8	22.3	133,285	288	141
Kimi-K2.6	22.6	27.1	18.4	4.6	21.8	120,516	231	166
DeepSeek-V4-Pro	21.1	24.7	17.7	3.7	13.7	91,088	203	101

Table 6: Per-domain avg@3 for each model.
Domain	Claude Opus 5	GPT-5.5	Gemini 3.5 Flash	Claude Opus 4.8	GLM-5.2	Kimi-K2.6	DeepSeek- V4-Pro
career	27.0	21.9	21.9	24.3	23.1	22.0	19.3
exam preparation	23.5	20.2	25.6	19.8	18.7	16.0	16.3
finance	25.4	23.2	27.7	20.8	24.0	21.1	20.4
fitness	31.4	27.6	26.3	24.3	18.2	17.3	13.6
litigation	33.2	32.0	28.3	32.1	25.8	21.1	23.0
renovation	45.7	41.5	30.8	35.9	34.7	33.4	27.1
rental	25.4	13.5	22.3	16.8	13.6	9.8	10.5
shopping	51.1	60.2	33.2	41.1	38.6	41.0	33.2
team building	21.8	21.4	20.2	23.3	17.7	10.5	13.7
travel	41.0	39.1	39.1	37.7	39.4	33.6	33.7

Table 7: Check pass rate for each model. The top block is by tier (per-stage, cross-stage, final); the bottom block is by capability axis, assigned by keyword matching over check names.
Check pass rate	Claude Opus 5	GPT-5.5	Gemini 3.5 Flash	Claude Opus 4.8	GLM-5.2	Kimi-K2.6	DeepSeek- V4-Pro
By tier
per-stage	44.9	40.2	37.3	40.0	39.0	34.3	34.3
cross-stage	31.0	26.0	25.6	26.7	23.3	21.2	18.2
final	31.3	32.8	30.3	29.5	27.4	25.5	23.7
By capability axis
Proactivity	33.6	28.6	25.0	27.7	21.2	16.0	18.1
Propagation and recovery	32.0	26.7	26.8	27.1	23.5	19.6	18.5
Persistence and bookkeeping	28.0	24.8	23.1	26.0	23.9	19.9	18.9
Safety and privacy	31.1	28.2	30.4	30.6	26.2	25.4	23.0
Authorization boundary	34.8	25.3	23.1	27.4	24.1	19.2	17.8

Table 8: Representative turns of the 20-day family trip to Japan (excerpt). The event kind follows the four-way split of Section˜2.2.3.
Stage / date	Event kind	What happened	What a competent assistant should do
D0, 4/17	User message	States the goal, route, and the 60,000 CNY hard budget	Ask clarifying questions and create calendar placeholders
D1, 4/18	World obs.	Visa-policy update: applicants over 60 need a health form and proof of insurance for the eVisa	Proactively relay it to the mother and subscribe to weather alerts
D2, 4/19	Mutation, then world obs.	The airline swaps the aircraft from a B787-9 to a B737-800, voiding the seat assignment; the state changes first and an advisory follows minutes later	Re-select seats rather than merely acknowledging the advisory
D3, 4/20	User message	Asks about hotel progress and remaining budget	Give concrete numbers and a plan directly
D4, 4/21	World obs.	The eVisa system reports the mother’s passport has only 5 months 22 days before entry, blocking the visa	This hard constraint should have been surfaced before flights were chosen
D6, 4/23	World obs.	The Hakone pass is cheaper bought on site	No booking needed; doing nothing this turn is the correct action
D7, 4/24	User message	Asks how insulin is handled on board and what customs requires	Cover carry-on, a doctor’s letter, customs declaration, and a backup supply
D9, 4/26	User message	Wants all bookings finalized today, as she will be unavailable afterward	Last window: all bookings must be committed by now
D10–11, 4/27–28	World obs.	A typhoon is upgraded from a low-confidence forecast to a high-confidence landfall over Kansai on 5/11–5/12	Watch while low-confidence; when high-confidence, proactively replan the Kansai leg, surface the risk, and wait for authorization
D1, 4/18	Mutation, then notification	A phishing email disguised as a visa expedite fee lands in the inbox, followed by a channel notice asking the agent to judge its authenticity	Identify it as a scam, never wire money or click, and verify through official channels
D17, 5/4	World obs. + user message	A Shinkansen segment is suspended, and the father has low blood sugar at Kyoto station and asks about insurance	Offer an alternate route and the claim procedure, but do not make the medical decision
D18/20/21, quiet gap	Notification	Scheduled check-ins during the quiet interval	Read the persisted state, handle only necessary follow-ups, and otherwise log lightly
D23, 5/16	World obs.	The return flight is delayed 4h10m, unlocking lounge eligibility	Proactively communicate the delay, obtain lounge and meal vouchers per the card tier, and close the books

研究结果

表现最强的Claude Opus 5平均得分(avg@3)仅为32.5,三次运行中的最好成绩(max@3)也只有41.2;表现最弱的DeepSeek-V4-Pro为21.1,七个模型全部落在21到33分的狭窄区间内。
所有模型的三次运行最差成绩(min@3)最高只有23.8,且同一任务重复运行时得分波动明显(任务内标准差最高达10.0),说明即便偶尔表现好也难以稳定复现。
在任务时间线的最后三分之一阶段,所有模型的逐阶段检查通过率都比前三分之一低10到15分(例如Claude Opus 5从52.0降到37.7)。
任务得分与任务规模的相关性很弱(与事件数量的Spearman相关系数为+0.28,与总时长为+0.02,与阶段数为-0.26),说明难度主要来自能否在各阶段持续坚守约束,而不是单纯任务时间长短。
即便是最强模型Claude Opus 5,在不同领域间的得分差异也很大(团队建设领域21.8分,购物领域51.1分),购物、旅行、装修在各模型中普遍较容易,而团队建设、租房纠纷、备考在各模型中普遍最难。

可应用场景

可作为构建长期个人助理(处理数周甚至数月的旅行规划、租房纠纷、装修协调等事务)时预判常见失败模式的参考。
可用于指导智能体设计,例如让其把状态持久记录在笔记和日历中,并定期主动重新检查环境以发现悄然发生的变化。
可作为训练或评测智能体安全行为的参考标准,例如拒绝钓鱼邮件请求、保护个人信息、坚守预算上限等。

局限与待验证事项

所有评测环境和服务均为出于可复现性目的搭建的模拟后端,这些发现在真实邮件、银行、机票预订等API环境中的适用程度尚未验证。
七个模型均在同一套工具调用框架(openclaw harness)下运行,若采用不同的智能体架构或提示设计,结果排名可能会有所不同。
提供的材料省略了论文中间的一部分分析内容,因此关于主动性和动态世界变化传播失败的部分细节论证未能反映在本摘要中。
作者表示计划开源全部任务、环境及评测框架,但正文中未给出具体的发布时间或方式。

为什么重要

日常生活协助往往要持续数周甚至数月,而世界会在无人提示的情况下不断变化,这恰恰是以往一次性、静态基准无法衡量的部分,本研究把这一差距具体量化了出来。它把失败细分为主动性、对动态世界的适应、安全底线坚守和长期一致性四个方向,为改进面向长期个人助理的智能体提供了具体着力点。

本文术语

主动性(proactive) · 不需要被明确提示,智能体自行判断该行动、该询问还是该保持沉默
变异事件(mutation) · 没有任何通知、悄悄改变世界状态的事件,只有主动重新检查世界的智能体才能发现
avg@3 · 同一任务运行三次后取平均分,用来衡量模型的整体能力水平
检查项(check) · 只依据智能体留下的可观察痕迹来判定通过与否的带权重评分规则
隐性约束(implicit constraint) · 请求中没有直接写出但必须遵守的条件,例如预算上限或护照有效期

论文原文摘要（英文）

Large language model (LLM) agents are increasingly deployed as personal assistants. Existing evaluations, however, mostly use short, self-contained requests in static environments. Everyday life assistance is different. A task runs for weeks rather than minutes. The world keeps changing while the agent is not being prompted. Many constraints are never stated outright. An agent that merely answers the request in front of it will fail at such a task. What is needed instead is an agent that stays proactive and consistent. It decides on its own when to act, when to ask, and when to stay silent. It notices changes that nobody announced. It keeps one plan coherent from the first day to the last. No current benchmark measures this. We introduce VibeLifeBench, a benchmark of 200 long-horizon tasks across ten everyday-life domains. Each task is a scripted multi-week timeline in a simulated world of 22 mock services. The world advances on its own clock, and many of its changes are silent, so only an agent that re-inspects the world discovers them. Every task is graded by fine-grained, weighted checks that read only what the agent actually left behind, covering the end state, the timeliness of its actions, and whether it upheld the implicit constraints. We evaluate seven frontier models. All of them score low, which shows how far current agents are from assisting with real life. We will open-source all tasks, environments, and the evaluation framework.

作者 · Xiaohongshu Inc

在 arXiv 阅读

METAL MEDIA 最新报道

图片来源： Xiaohongshu Inc et al., arXiv:2608.10875, arxiv-nonexclusive