The Inference Report

The SWE-rebench rankings show no movement from the prior cycle, with OpenAI's gpt-5.5-2026-04-23-xhighModel holding 62.7% plus-or-minus 0.91% at the top, followed by JunieJunieAgent at 61.6% and OpenAICodexAgent at 60.4%. The Artificial Analysis benchmark, by contrast, absorbed a substantial reshuffle across its 399 entries: Grok 4.5 entered the list at number four with a score of 53.8, displacing Claude Opus 4.7 and pushing the prior top-ten models down one position each, while DeepSeek V3.1 Terminus climbed from position 89 (26.3) to position 79 (30.4), a gain of 4.1 points that stands out among the mid-tier movers, and several new entries like Mistral Small 3.1 and Gemma 3n E4B Instruct Preview appeared in the list without prior ranking data. The SWE-rebench methodology remains opaque in terms of evaluation scope and error bars, though the confidence intervals reported suggest reasonable precision; the Artificial Analysis scores, lacking error bounds, offer no such transparency about measurement reliability or sample size. Neither benchmark's movement tells a coherent story of progress: SWE-rebench's flatness could reflect either genuine plateauing or evaluation drift, while Artificial Analysis's churn across hundreds of models without clear methodology or temporal consistency makes it difficult to distinguish real capability shifts from ranking noise. The two benchmarks diverge substantially in their top rankings, with SWE-rebench dominated by agent-based systems and OpenAI variants while Artificial Analysis favors Anthropic's Claude models, suggesting they measure different problem domains or that one systematically advantages certain architectural choices over others.

Cole Brennan