The Inference Report

The SWE-rebench rankings show minimal movement at the top tier, with gpt-5.5-2026-04-23-xhigh holding 62.7% and Codex at 60.4%, but significant volatility below position five signals instability in how models perform on this coding task. Gemini 3.1 Pro Preview dropped from 57.2% on Artificial Analysis to 51.1% on SWE-rebench, falling from fourth to tenth place, while Kimi K2.6 fell from 53.9% to 46.5% and GLM-4.7 declined from 42.1% to 38.2%, suggesting these models may not generalize equally across different coding benchmarks or that SWE-rebench applies stricter evaluation criteria. Conversely, GLM-5.1 held relatively steady between 50.7% and 51.4%, and Claude models maintained consistent rankings across both benchmarks, indicating more reliable performance on code generation tasks. The divergence between SWE-rebench and Artificial Analysis rankings below 50% raises questions about benchmark design: SWE-rebench appears to penalize certain architectural approaches more heavily, or the two evaluations measure meaningfully different aspects of coding capability. Without access to SWE-rebench's methodology documentation, the 5-7 point gaps between benchmark results for the same models cannot be attributed definitively to task difficulty, evaluation harshness, or genuine capability differences, making it premature to treat either ranking as a complete picture of coding performance.

Cole Brennan