The Inference Report

The SWE-rebench leaderboard shows stasis at the top with Claude Code holding 52.9% and Junie at 52.1%, while significant reshuffling occurs in the middle tiers. Claude Opus 4.5 dropped from position 8 with 49.7% to position 12 with 43.8% on Artificial Analysis, a substantial decline that warrants scrutiny of whether the evaluation methodology shifted or if the model's capabilities genuinely regressed on this particular task distribution. Conversely, Kimi K2 Thinking climbed from position 28 with 40.9% to position 13 with 43.8% on Artificial Analysis, suggesting either improved inference or a benchmark revision that favors its approach. The Gemini 3 Pro Preview moved from position 11 at 48.4% to position 8 at 46.7% on SWE-rebench, a modest decline consistent with natural variance, though the divergence between the two benchmarks (Artificial Analysis shows it at 48.4%) hints at methodological differences in how they score the same model outputs. GLM-5 dropped from position 7 with 49.8% on Artificial Analysis to position 15 with 42.1% on SWE-rebench, a 7.7-point gap that is difficult to attribute to random noise and suggests these benchmarks may be testing different aspects of code generation capability. The lack of movement in the top five positions on SWE-rebench combined with large swings in the 7-20 range indicates the benchmark is sensitive enough to detect real differences but that the frontier models have plateaued relative to their challengers, a pattern worth monitoring across future cycles to determine whether we are seeing genuine convergence or measurement instability.

Cole Brennan