ScholarPulse 日报 2026-07-17

2026-07-17 学术简报：2 篇。论文提出成本-成功评估框架，主张安全代理评估需兼顾经济效率与操作适应性，而非仅关注成功率，以反映实际安全操作中的预算约束。

今日速览

序号	标题	来源	日期	主题	推荐等级
1	Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents	arXiv	2026-07-16	AI-Agent	高
2	Can We Trust Item Response Theory for AI Evaluation?	arXiv	2026-07-16	RAG	高

重点论文与技术动态

1. Beyond Success Rate: Cost-Aware Evaluation of Offensive and Defensive Security Agents

来源：arXiv
日期：2026-07-16
作者/机构：Paul Kassianik, Blaine Nelson, Yaron Singer
主题标签：AI-Agent, arXiv
推荐等级：高
分类：cs.CR, cs.AI

一句话结论

论文提出成本-成功评估框架，主张安全代理评估需兼顾经济效率与操作适应性，而非仅关注成功率，以反映实际安全操作中的预算约束。

核心内容

进攻性CTF任务性能随测试时计算增加而提升，开源模型可接近专有系统且保持成本竞争力。
防御性SOC调查任务成功更依赖工具使用规范、遥测导航和选择性增强，而非单纯推理预算。
安全代理基准测试应整合经济效率、操作适配性与任务成功率，避免仅聚焦峰值性能。

方法与数据

评估在进攻性Cybench挑战和防御性Splunk BOTS v1调查挑战上进行。
摘要未明确数据集规模或具体参数细节。

价值判断

值得关注：成本-aware评估为安全团队选择实用代理提供清晰依据，避免高成本低效方案。
可复用点：该框架可推广至其他安全代理基准测试，推动行业标准化评估实践。
局限/待核查：摘要未提及模型局限或待验证点。

摘要

Security-agent evaluations commonly measure peak offensive capability under generous inference budgets, emphasizing vulnerability discovery, exploit development, penetration testing, and CTF completion. Such measurements are useful but incomplete: in operational security, every reasoning step, tool call, telemetry query, and enrichment request consumes budget. We evaluate language-model security agents through this cost-success lens on offensive Cybench challenges and defensive Splunk BOTS v1 investigation challenges. Instead of reporting only best-case success, we compare models at fixed cost levels and decompose performance by inference spend and tool spend. Our results show distinct scalingregimes for red- and blue-team tasks. Offensive CTF performance improves with additional test-time compute, and scaled open-weight models can approach frontier proprietary systems while remaining cost-competitive. Defensive SOC investigation does not scale in the same way: success depends more heavily on disciplined tool use, telemetry navigation, and selective enrichment than on raw reasoning budget alone. We argue that security-agent benchmarks should measure economic efficiency and operational fit alongside task success. Cost-aware, SOC-native evaluations provide a clearer picture of which models are practically useful today and where defensive agents still need to improve. We present an interactive website with our results https://evals.frontier.security.

2. Can We Trust Item Response Theory for AI Evaluation?

来源：arXiv
日期：2026-07-16
作者/机构：Han Jiang, Sunbeom Kwon, Jinwen Luo, Ziang Xiao, Susu Zhang
主题标签：RAG, arXiv
推荐等级：高
分类：cs.AI

一句话结论

研究发现，AI基准测试中项目反应理论（IRT）的可靠性因数据特性与人类测试不匹配而受限，经典估计器在大型设置中不可行，可扩展估计器在小或非正态分布模型集上导致不可靠推断。

核心内容

AI基准数据通常涉及较少评估模型、更多项目，且能力分布可能偏斜、聚类或多重模态，与IRT设计初衷不符。
通过18,000种模拟条件系统评估了四种IRT估计工具（边际最大似然、马尔可夫链蒙特卡洛、变分推断、神经伪孪生估计器）的计算可行性与推断可靠性。
结果表明，经典估计器在大型基准中不可行，而可扩展估计器在小模型集或非正态能力分布下产生不可靠的项目级和排名推断。

方法与数据

基于六种广泛LLM基准的参数模拟响应矩阵。
摘要未明确具体估计工具细节。

价值判断

值得关注：揭示了IRT在AI评估中的潜在风险，警示基准测试设计需考虑数据特性以避免扭曲结论。
可复用点：模拟方法和18,000种条件评估框架可直接应用于其他AI评估场景。
局限/待核查：在小模型集或非正态能力分布下，IRT推断可靠性显著降低，需特定样本大小和诊断保障可信度。

摘要

AI benchmarks increasingly leverage item-level statistical models, particularly item response theory (IRT), to estimate model capabilities, rank systems, select informative examples, and diagnose benchmark quality. However, AI benchmark data often departs from the data regime of human testing, for which standard IRT estimation tools were originally developed: benchmarks typically involve fewer evaluated models, far more items, and capability distributions that may be skewed, clustered, or multimodal. We examine how these regime mismatches challenge the reliability of IRT modeling for AI evaluation. Using item parameters and capability distributions derived from six widely used LLM benchmarks, we simulate response matrices under three common IRT models and compare four estimation tools used in recent benchmark studies: marginal maximum likelihood, Markov chain Monte Carlo, variational inference, and a neural pseudo-Siamese estimator. Across 18,000 simulation conditions, we systematically evaluate computational feasibility, scalability, and the reliability of IRT inferences about model rankings, predicted performance, and item characteristics. Results show that classical estimators can become infeasible in large benchmark settings, whereas scalable estimators can produce unreliable item-level and ranking inferences with small or nonnormally distributed model sets. This study identifies when latent trait models reliably support or risk distorting AI benchmarking claims, and what sample sizes and diagnostics are needed for trustworthy use.