The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, a 12.3 percentage point gain from its previous ranking of 53 on Artificial Analysis, while gpt-5.2-2025-12-11-medium sits at 64.4% and GLM-5 climbed to 62.8% from 49.8 on the older benchmark. The movement reflects a fundamental divergence between the two evaluation frameworks: SWE-rebench appears to measure software engineering capability through a different task distribution or methodology than Artificial Analysis, producing a reshuffling that extends beyond the top tier. Gemini 3.1 Pro Preview dropped from #2 on Artificial Analysis (57.2) to #5 on SWE-rebench (62.3), while Kimi K2.5 jumped from #20 (46.8) to #13 (58.5), and Kimi K2 Thinking advanced from #42 (40.9) to #17 (57.4), suggesting these models perform meaningfully better on the specific coding problems SWE-rebench isolates. The Artificial Analysis benchmark may weight different problem categories or evaluation criteria, or SWE-rebench may employ stricter pass conditions, but without documentation of the methodological differences between these frameworks, the magnitude of score inflation across the board (top models gaining 8-12 points) raises questions about whether the benchmarks are measuring comparable constructs or if one applies easier evaluation criteria. The consistency of Claude's dominance across both rankings and the clustering of multiple models in the 58-62% band on SWE-rebench suggests the benchmark has resolution to differentiate models, but direct score comparison between the two systems is unreliable.

Cole Brennan