The Inference Report

The two benchmark systems show stability at the top tier but divergence in methodology that warrants scrutiny. On SWE-rebench, the leader remains OpenAI gpt-5.5-2026-04-23-xhighModel at 62.7% ± 0.91%, with positions two through ten unchanged from the prior snapshot: JunieJunieAgent (61.6%), OpenAICodexAgent (60.4%), AnthropicClaude CodeAgent (59.6%), and five others holding their ranks. The confidence intervals are tight enough that these orderings reflect real performance separation, though the 1.1-point gap between first and second suggests the frontier remains contested. Artificial Analysis data tells a different story about which models matter. Claude Opus 5 leads there at 60.7, ahead of Claude Fable 5 (59.9) and GPT-5.6 Sol (58.9), a ranking that clusters Anthropic and OpenAI variants rather than privileging agent-based configurations. The two datasets agree on little: SWE-rebench elevates specialized agent frameworks (JunieAgent, Cursor) to positions two and nine, while Artificial Analysis buries agent variants in the mid-tier or lower. This gap reflects fundamental differences in evaluation scope. SWE-rebench appears narrowly calibrated to software engineering tasks with reproducible metrics and error bars; Artificial Analysis spans broader capability assessment and lacks reported confidence bounds, making score movements harder to interpret. Neither system has shifted meaningfully since the previous snapshot, suggesting either genuine stability or evaluation lag. The practical signal is clearest on SWE-rebench's top twenty, where the spread from 62.7% to 38.4% separates systems with meaningfully different engineering performance. Below that, the Artificial Analysis tail extends to models scoring 1.0, a floor that lacks discriminative power and suggests saturation in the evaluation methodology rather than genuine parity.

Cole Brennan