The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, a gain of 12.3 percentage points from its previous ranking of 53 on Artificial Analysis, though these are distinct benchmarks measuring different problem sets and should not be directly compared as improvement on the same task. The SWE-rebench leaderboard shows tighter clustering in the upper tier, with models ranked 2 through 7 all scoring between 64.4% and 62.3%, suggesting convergence in code-solving capability among leading systems. GLM-5 and GLM-5.1 both advanced significantly on Artificial Analysis, moving from positions 17 and 14 to 49.8 and 51.4 respectively, while Kimi K2 Thinking jumped from 54 to 40.9, indicating Chinese-developed models are narrowing the gap. Gemini 3.1 Pro Preview declined from 57.2 to 57.2 on Artificial Analysis (no change) but appears at position 7 on SWE-rebench with 62.3%, down from an implied higher position previously. The SWE-rebench methodology evaluates code agents on real GitHub issues requiring multi-step reasoning and tool use, while Artificial Analysis covers general reasoning tasks, so raw score differences across benchmarks reflect task difficulty rather than model capability regression. Within SWE-rebench, the spread from position 1 to position 10 spans only 4.3 percentage points, suggesting marginal gains require increasingly refined approaches rather than architectural leaps. The Artificial Analysis rankings show broader stratification, with positions 1 through 10 spanning 7.1 points, indicating that general reasoning benchmarks may be more discriminative at the frontier than specialized coding benchmarks at present.

Cole Brennan