The Inference Report

The SWE-rebench leaderboard shows no movement from the previous snapshot: OpenAI's gpt-5.5-2026-04-23-xhigh model holds first at 62.7%, followed by JunieAgent at 61.6% and OpenAI CodexAgent at 60.4%, with confidence intervals tight enough to distinguish genuine separation between top performers. The Artificial Analysis benchmark presents a different picture, one of modest shuffling rather than substantive reordering. Claude Fable 5 leads at 59.9, Claude Opus 4.8 sits at 55.7, and GPT-5.5 ranks third at 54.8, but the list contains no new entries and the scoring appears identical to prior rankings. Two minor position swaps occur in the mid-range: Qwen3 32B moves from rank 217 to 209 with an improvement from 10.5 to 11.5, and Sarvam 105B and Magistral Small 1.2 exchange positions around rank 204-205 without score changes, suggesting database reorganization rather than actual performance shifts. The methodology underlying both benchmarks remains opaque. SWE-rebench reports confidence intervals, which implies repeated trials or cross-validation, yet no detail appears on the evaluation protocol, task distribution, or whether results are deterministic across runs. Artificial Analysis provides no uncertainty quantification whatsoever, making it impossible to assess whether fractional score differences reflect genuine capability gaps or measurement noise. The two benchmarks diverge substantially at the top (gpt-5.5-xhigh leads SWE-rebench but ranks third on Artificial Analysis), raising questions about whether they measure the same construct or whether one dataset better captures real-world code repair needs. Without clarification of what each benchmark tests, how tasks are sampled, and whether scoring is reproducible, the rankings function as indices rather than measures of engineering competence.

Cole Brennan