The Inference Report

The SWE-rebench rankings remain stable at the top tier, with gpt-5.5-2026-04-23-xhigh holding first place at 62.7%, Codex at 60.4%, and Claude Code at 59.6%, indicating that the highest-performing systems have consolidated their positions. Movement occurs in the middle ranks where Gemini 3.1 Pro Preview dropped from fourth to tenth place on SWE-rebench (57.2% to 51.1%), a 6.1-point decline that signals either a methodological shift or genuine regression in this model's code-solving capability. On the Artificial Analysis benchmark, the top tier similarly stabilizes with Claude Opus 4.8 leading at 61.4 and GPT-5.5 at 60.2, though the broader ranking reveals substantial churn below the top ten: Qwen3.7 Max enters at sixth place (56.6), while older GPT versions and specialized models shuffle downward. GLM-4.7 shows the most striking movement, rising from forty-eighth to forty-ninth on Artificial Analysis but falling on SWE-rebench from 38.2% to 42.1%, a pattern suggesting the benchmarks measure different problem distributions. Kimi K2.6 declined notably from eighth to thirteenth on SWE-rebench (53.9% to 46.5%), a 7.4-point drop that warrants scrutiny into whether the evaluation protocol changed or the model's inference behavior shifted. The divergence between SWE-rebench and Artificial Analysis rankings for mid-tier models like GLM-5.1 (11th on SWE-rebench at 50.7%, 18th on Artificial Analysis at 51.4%) suggests these benchmarks are not measuring identical capabilities, likely because SWE-rebench emphasizes repository-level problem solving while Artificial Analysis may weight different code-generation tasks. Without historical Artificial Analysis data from a prior snapshot, the stability of that leaderboard's top positions appears genuine rather than volatile, though the accumulation of new entries like Qwen3.7 Plus at eleventh place indicates the benchmark's sample is expanding rather than consolidating.

Cole Brennan