The Inference Report

The SWE-rebench coding benchmark shows no movement from the previous snapshot: OpenAI's gpt-5.5-2026-04-23-xhighModel remains at 62.7% ± 0.91%, followed by JunieJunieAgent at 61.6% ± 0.64% and OpenAI CodexAgent at 60.4% ± 1.37%. The top six positions are occupied by the same models in identical order, with confidence intervals wide enough that the observed differences could reflect measurement noise rather than genuine capability gaps. Anthropic's Claude CodeAgent holds fourth place at 59.6% ± 1.98%, a margin of 2.8 percentage points below the leader but with substantial uncertainty. The Artificial Analysis benchmark, which samples a far larger model population (398 entries versus 24), tells a different story: Claude Fable 5 leads at 59.9, followed by Claude Opus 4.8 at 55.7 and GPT-5.5 at 54.8. This ranking divergence between the two benchmarks reflects fundamental methodological differences: SWE-rebench appears to prioritize agent-based systems and their specific configurations, while Artificial Analysis weights base model performance across a broader set of evaluation conditions. The SWE-rebench results are methodologically cleaner for comparing agentic approaches, but the stability of the top tier and the width of confidence bands suggest the benchmark may lack sensitivity to distinguish performance in the 50-65% range, where incremental improvements would be most valuable for practical software engineering tasks.

Cole Brennan