今日速览
| 序号 | 标题 | 来源 | 论文日期 | 主题 | 推荐等级 |
|---|---|---|---|---|---|
| 1 | SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model | arXiv | 2026-06-12 | RAG | 中 |
| 2 | StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance | arXiv | 2026-06-12 | AI-Agent | 高 |
重点论文与技术动态
1. SIMMER: Benchmarking Latent Failures in LLM Executable Planning with a World Model
- 来源：arXiv
- 论文日期：2026-06-12
- 作者/机构：Xiaoxin Lu, Ranran Haoran Zhang, Rui Zhang
- 主题标签：
RAG,arXiv - 推荐等级：中
- 分类：cs.CL, cs.AI
一句话结论
SIMMER基准测试在家庭环境中揭示LLM规划中潜在故障率高达56%，且反事实推理可降低72%潜在故障和75%不可逆案例。
核心内容
- 潜在故障不同于立即失败，不立即中断执行，但会无声损害目标达成，严重时导致不可逆伤害。
- SIMMER基于厨房领域人工策划的符号世界模型，包含77动作、262物体及约46,800种语义交互，源自真实烹饪脚本。
- 实验显示，六种LLM中前沿模型仅17%计划无错误，56%含潜在故障，多数致不可逆后果。
方法与数据
- 采用状态机执行器验证计划，检测即时前提违反、潜在危害和不可逆故障。
- 世界模型基于真实烹饪脚本构建，确保交互语义真实。
价值判断
- 值得关注：现有基准忽视潜在故障，易引发安全风险。
- 可复用点：反事实推理模拟可显著减少潜在故障率。
- 局限/待核查：摘要未明确提及局限。
摘要
Large language models (LLMs) are increasingly deployed as planners for autonomous agents in household environments. While existing benchmarks evaluate whether LLM-generated plans execute successfully, they overlook a critical type of failure: latent failures. Unlike immediate failures that trigger instant feedback at execution time and enable timely correction, latent failures do not immediately halt plan execution but silently compromise goal achievement. In severe cases, they cause irreversible harm. To address this gap, we introduce SIMMER, a benchmark for evaluating latent failures in LLM planning through a human-curated symbolic world model grounded in the kitchen domain. SIMMER defines a world model comprising 77 actions, 262 unique objects, and approximately 46,800 possible interactions that are semantically realistic, derived from real-world cooking scripts. It then leverages a state machine executor that validates plans against the world model and detects immediate precondition violations, latent hazards, and irreversible failures. Experiments across six LLMs show that even frontier models achieve at most 17% error-free plans. Moreover, up to 56% of plans contain latent failures, the majority of which lead to irreversible consequences. We further demonstrate that explicit state reasoning via counterfactual foresight simulation can reduce latent failures by up to 72% and irreversible cases by up to 75%, suggesting a promising direction for more robust LLM planners.2. StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistance
- 来源：arXiv
- 论文日期：2026-06-12
- 作者/机构：Guanming Liu, Yuqi Ren, Hansu Gu, Peng Zhang, Weihang Wang, Jiahao Liu 等 8 人
- 主题标签：
AI-Agent,arXiv - 推荐等级：高
- 分类：cs.AI
一句话结论
当前代理记忆系统在利用观察证据和将反馈转化为可靠后续行为方面表现不佳，即使证据被存储或反馈被局部整合。
核心内容
- 现有记忆基准仅孤立测试对话回忆或任务改进，未覆盖从流式观察到后续辅助的连续轨迹。
- StreamMemBench通过EgoLife egocentric streams的证据锚点构建两步任务序列，使用四个指标（证据回忆、初始证据使用、反馈整合、后续重用）评估系统表现。
- 实验显示，八种记忆系统在两个骨干模型上测试，常失败于使用观察证据或转化反馈，即使证据存储或反馈局部整合。
方法与数据
- 任务序列基于EgoLife egocentric streams构建。
- 摘要未明确数据集规模等细节。
价值判断
- 值得关注：首次系统评估流式记忆在面向未来辅助中的作用，填补现有基准空白。
- 可复用点：基准工具开源于GitHub，可直接用于其他代理记忆系统测试。
- 局限/待核查：摘要未明确具体局限。