The Inference Report

Claude Opus 4.6 climbed from eighth to first on SWE-rebench, moving 12.3 percentage points from 53% to 65.3%, a movement that reshuffles the entire coding benchmark landscape but warrants scrutiny on whether the test itself remained stable or if the evaluation methodology changed. The top tier has tightened considerably: gpt-5.2-2025-12-11-medium sits at 64.4%, GLM-5 and Junie both at 62.8%, and gpt-5.4-2026-03-05-medium holding steady at 62.8%, creating a compressed band where fractional improvements matter. Below the top five, the ranking has reordered substantially; Gemini 3.1 Pro Preview dropped from third to seventh despite scoring 62.3%, while several models gained ground including GLM-5 (from rank 16 at 49.8% to rank 3 at 62.8%), Kimi K2.5 (from rank 28 at 46.8% to rank 16 at 58.5%), and Kimi K2 Thinking (from rank 53 at 40.9% to rank 21 at 57.4%), suggesting either substantial capability improvements across Chinese models or a shift in benchmark composition favoring their training distribution. On Artificial Analysis, the rankings show minimal movement in the top tier with GPT-5.5 still leading at 60.2 and Claude Opus 4.6 now ninth at 53, a nine-point gap that contradicts the SWE-rebench clustering and raises questions about benchmark alignment. Grok 4.3 entered the Artificial Analysis top 100 at position eight with 53.2, while most other models maintained their prior positions, suggesting this benchmark is more stable but possibly measuring a different capability or using different evaluation criteria. The divergence between SWE-rebench's dramatic reshuffling and Artificial Analysis's relative stability indicates these benchmarks are not measuring the same problem space, or that one has undergone methodological revision without documentation.

Cole Brennan