The Inference Report

The SWE-rebench and Artificial Analysis benchmarks show minimal movement at the top tier, with Claude Code holding 52.9% on SWE-rebench while the next five models cluster within 0.8 percentage points, a pattern that has held steady. At Artificial Analysis, GPT-5.4 and Gemini 3.1 Pro Preview both score 57.2%, with the third-place GPT-5.3 Codex at 54, but the rankings below position 10 reveal more volatility: Kimi K2 Thinking jumped from position 30 to 13 on SWE-rebench with a 2.9-point gain to 43.8%, while Claude Opus 4.5 dropped from position 8 to 12 on SWE-rebench, losing 5.9 points to 43.8%, and GLM-5 fell from position 7 to 15 on SWE-rebench, declining 7.7 points to 42.1%. On Artificial Analysis, most entries shifted by single positions rather than substantial score changes, with new entries like MiniMax-M2.7 (49.6), GPT-5.4 mini (48.1), and GPT-5.4 nano (44.4) appearing in the upper rankings and Sarvam 105B (18.2) and Sarvam 30B (12.4) entering further down. The SWE-rebench benchmark's methodology remains unclear from the data provided, limiting assessment of whether these movements reflect genuine capability differences or testing variance; the compression of scores in the 40-52% range on SWE-rebench suggests either a ceiling effect or a narrowing gap between frontier models, while Artificial Analysis's broader distribution and deeper ranking list indicate different evaluation criteria or model coverage.

Cole Brennan