The Inference Report

The SWE-rebench rankings show no movement from the previous snapshot, with OpenAI gpt-5.5-2026-04-23-xhighModel holding position one at 62.7% ± 0.91% and the top twenty entries unchanged in both order and score. Artificial Analysis data, however, reveals substantial repositioning across its 404-model roster, with Claude Fable 5 now leading at 59.9 (up from Claude Opus 4.8's prior prominence), GPT-5.6 Sol at 58.9, and Claude Opus 4.8 dropping to third at 55.7. The divergence between benchmarks raises questions about methodology: SWE-rebench measures software engineering task completion with confidence intervals tight enough to suggest controlled evaluation conditions, while Artificial Analysis scores lack published error margins and show model rankings that sometimes contradict SWE-rebench orderings (DeepSeek V4 Pro scores 44.3 on Artificial Analysis but 42.7% on SWE-rebench, for instance). The stability of SWE-rebench's top tier contrasts sharply with Artificial Analysis volatility, where mid-tier models shuffle frequently and smaller models cluster at single-digit scores with minimal differentiation. Without documentation of Artificial Analysis methodology, sample sizes, or evaluation protocols, the significance of its reordering remains unclear; the SWE-rebench data, by contrast, permits confidence in its findings because the uncertainty bands are explicit and the evaluation domain (code generation for real repository issues) is concrete and reproducible.

Cole Brennan