The Inference Report

Claude Opus 4.6 moved from fourth to first on SWE-rebench, improving from 53% to 65.3%, a 12.3-point gain that positions it ahead of gpt-5.2-2025-12-11-medium at 64.4%. GLM-5 climbed from tenth to third, jumping 13 points to 62.8%, matching gpt-5.4-2026-03-05-medium. Gemini 3.1 Pro Preview fell from first to fifth despite scoring 62.3%, suggesting the benchmark may have tightened or other models benefited from architectural changes rather than uniform improvement across the field. Kimi K2.5 advanced from twentieth to thirteenth with a 11.7-point increase to 58.5%, and Kimi K2 Thinking jumped from forty-second to seventeenth with a 16.5-point gain to 57.4%. The Artificial Analysis rankings show minimal movement in the top tier, with most models holding their positions, suggesting SWE-rebench and Artificial Analysis are measuring different problem spaces or that SWE-rebench's methodology may be capturing recent architectural improvements that general-purpose benchmarks have not yet registered. The concentration of gains among Claude and Kimi variants, paired with Gemini's relative decline, hints at task-specific optimization rather than across-the-board capability expansion. Without methodological details on how SWE-rebench was constructed or modified, it is unclear whether these shifts reflect genuine progress on software engineering tasks or whether the benchmark itself has shifted in composition.

Cole Brennan