今日速览
| 序号 | 标题 | 来源 | 日期 | 主题 | 推荐等级 |
|---|---|---|---|---|---|
| 1 | Can AI agents conduct open-ended AI research? Early evidence from two case studies | arXiv | 2026-07-29 | AI-Agent | 高 |
| 2 | OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding | arXiv | 2026-07-29 | RAG | 高 |
重点论文与技术动态
1. Can AI agents conduct open-ended AI research? Early evidence from two case studies
- 来源：arXiv
- 日期：2026-07-29
- 作者/机构：Peter Kirgis, Sayash Kapoor, Andrew Schwartz, Stephan Rabanser, David Africa, Konstantinos Voudouris
- 主题标签：
AI-Agent,arXiv - 推荐等级：高
- 分类：cs.AI, cs.CY, cs.LG
一句话结论
当前AI代理能完成AI研究的工程任务，但无法实质性推进开放性研究问题。
核心内容
- 代理在shadow evaluations中完成工程无帮助，却无法推进NeurIPS 2026论文的核心研究问题，导致作者一致拒绝。
- 识别出五种失败模式：对可发表研究的判断失误、研究设计缺陷时缺乏创意、死胡同时无效回溯、资源意识不足及指令漂移。
- 通过第二个模型和支架的稳健性检查，失败模式被重复验证，证实了代理在研究决策中的系统性局限。
方法与数据
- 方法：代理承担高质量未发表论文的中心开放性问题，原作者评分（shadow evaluations）。
- 数据：基于两个NeurIPS 2026提交，代理获6天计算时间及数千美元资源。
价值判断
- 值得关注：揭示了AI代理工程能力与研究创新能力的差距，为自动化AI研究提供早期实证证据。
- 可复用点：shadow evaluations框架可标准化用于评估AI代理在开放性研究中的表现。
- 局限/待核查：代理在开放性任务中表现不足，需解决失败模式以提升研究决策能力。
摘要
Forecasts of explosive AI progress hinge on AI agents automating AI research. But evidence on whether agents can carry out open-ended AI research is thin. Current evaluations either test agents on narrow, verifiable tasks, which excludes open-ended research, or submit AI-generated papers to blind peer review, which is overstretched, stochastic, and suffers from poor review quality. We introduce a third way to measure progress towards AI R\&D automation. An agent takes on the central, open-ended research question of a high-quality unpublished paper, and the paper's original authors grade its output. We call these shadow evaluations. We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute. The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions. As a result, both papers were unambiguously rejected by the authors. We identify five recurring failure modes: poor judgment about the bar for publishable research, uncreative responses to shortcomings in the research design, ineffective backtracking from dead ends, poor resource awareness, and instruction drift. A robustness check with a second model and scaffold reproduced these failures. We release the expert reviews, survey responses, agent repositories, and logs. Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.2. OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding
- 来源：arXiv
- 日期：2026-07-29
- 作者/机构：Jingbo Zhou, Yusai Zhao, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao
- 主题标签：
RAG,arXiv - 推荐等级：高
- 分类：cs.AI, cs.CL, cs.HC
一句话结论
LLM代理在办公室任务中比人类更便宜快速，但交付质量尚未达到人类水平。
核心内容
- OmegaUse-OfficeVal基准包含100个办公室任务，源自从业者请求，平均需2.32小时人工完成，每个任务配对人工时间与任务价格代理。
- 经济信号支持直接比较人类成本与LLM推理成本，并实现价值加权评估，提升评估实用性。
- 评估显示所有LLMs显著更便宜快速，但交付质量未接近人类水平，与人类基线存在差距。
方法与数据
- 100个任务通过隐私保护过程适应，确保数据安全与隐私。
- 开发代码验证器基于细粒度标准，支持稳定评估。
价值判断
- 值得关注：经济信号使评估更贴近实际工作场景，支持成本与价值的量化比较。
- 可复用点：代码和数据集已开源，便于社区复用与扩展。
- 局限/待核查：LLM代理在任务完成质量上仍落后于人类，需进一步优化。