The Inference Report

Claude Opus 4.6 has consolidated the top position on SWE-rebench with 65.3%, climbing from #11 at 52.9% on Artificial Analysis, a gain of 12.4 percentage points that reflects either substantial model improvements or meaningful differences in how the two benchmarks evaluate code-solving capability. The broader SWE-rebench leaderboard shows clustering at the top: gpt-5.2-2025-12-11-medium, GLM-5, Junie, and gpt-5.4-2026-03-05-medium all sit within 1.6 points of each other between 62.8% and 64.4%, suggesting convergence among frontier models on this task. Notable climbers include GLM-5 (from #19 to #3, a 13-point jump), Kimi K2.5 (from #31 to #16, up 11.7 points), and Kimi K2 Thinking (from #56 to #21, up 16.5 points), indicating that Chinese-developed models have made tangible progress on repository-level code tasks. Gemini 3.1 Pro Preview declined from #3 to #7 on SWE-rebench while maintaining #3 on Artificial Analysis at 57.2, illustrating that benchmark choice materially affects perceived ranking. Claude Sonnet 4.6 moved from #14 to #9 on Artificial Analysis (51.7 to 60.7 on SWE-rebench), suggesting the models tested are stronger at the specific problem distributions in SWE-rebench than on Artificial Analysis's evaluation. The divergence between the two benchmarks raises a methodological question: SWE-rebench appears to emphasize end-to-end repository modification and integration, while Artificial Analysis may weight reasoning and breadth differently. Without access to the evaluation protocols themselves, the magnitude of these shifts makes it difficult to assess whether one benchmark has higher discriminative validity for production code work.

Cole Brennan