#evals
9 posts tagged with "evals".
-
Your Agent Catches Everyone's Mistakes But Its Own
• 5 min readNew research says self-correction fails because of the role label on the claim, not the claim's content. The fix is structural, and cheaper than you think.
-
Your Agent's Benchmark Score Is an Experiment, Not a Fact
• 6 min readRecent work shows a single agent leaderboard number is wrong three independent ways: it's noisy, it's overfit, and the judge measuring it is unreliable.
-
The Agent Doesn't Know When It's Failing
• 6 min readNew benchmarks measure calibrated refusal and premature self-stops, and the data says agent confidence signals are broken. Here's how to engineer around it.
-
State, Shells, and Shortcuts: The Agent Stack Spent Late May Fixing Its Foundations
• 6 min readMCP went stateless, a wave of coding-agent RCEs landed, and a new benchmark measured reward hacking — the three properties that make an agent useful all became liabilities.
-
Agents Are Writing Their Own Skills — and Retrieval Is the New Bottleneck
• 5 min readMay 2026's skill-library research shows agents can now accumulate reusable capabilities, but retrieving and adopting them is harder than generating them.
-
Capability Went Up. Reliability Didn't. That's the Agent Problem Now.
• 5 min readNew work argues agents are measured wrong: accuracy keeps climbing while consistency, robustness, and predictability barely move. The fix is architectural.
-
The Agent Benchmark Reckoning of May 2026
• 6 min readSTATE-Bench, DeepSWE, Agent Island, SWE-bench Live: a wave of new evals exposes how much the old leaderboards were inflating.
-
The Agent Is a Workload, Not a Script
• 6 min readMid-May 2026 quietly shipped the operations layer for agents — versioned environments, runtime drain, behavior-based evals, portable skills.
-
Agent Evals in 2026: Beyond LLM-as-Judge
• 10 min readVibes-based scoring is finally dying. Trajectory eval, rubric eval, golden replay, and the test pyramid that production agent teams actually run.
