The Inference Report

Claude Opus 4.6 moved from fourth to first on SWE-rebench with a 65.3% score, a 12.3-point gain over its previous 53%, while Gemini 3.1 Pro Preview dropped from the top ranking on Artificial Analysis to sixth on SWE-rebench despite maintaining 62.3%, suggesting the two benchmarks now diverge meaningfully in what they reward. The SWE-rebench leaderboard shows tighter clustering at the top, with positions two through five spanning only 1.6 percentage points, indicating reduced separation between leading models on coding tasks. Chinese models made notable gains on SWE-rebench: GLM-5 climbed from tenth to third (49.8 to 62.8%), Kimi K2.5 rose from twentieth to sixteenth (46.8 to 58.5%), and GLM-4.7 advanced from thirty-fourth to fourteenth (42.1 to 58.7%), while on Artificial Analysis the top tier remains dominated by Anthropic and OpenAI variants, with Claude Opus 4.7 newly entering at first place and Gemini 3.1 Pro Preview sliding to second. The Artificial Analysis benchmark shows minimal absolute movement across most positions, with entries reordering but scores remaining largely stable, whereas SWE-rebench displays larger score inflation across the board, raising questions about whether the benchmarks are measuring consistent capabilities or whether SWE-rebench's evaluation methodology has shifted. JT-MINI dropped entirely from Artificial Analysis rankings after placing at 109 with 25.4 points, but no corresponding SWE-rebench removal is documented, leaving unclear whether this reflects model discontinuation or benchmark revision. The divergence between these two evaluation frameworks is now pronounced enough to warrant scrutiny of their test construction: if both measure code generation ability, the gap between Gemini's ranking (first on Artificial Analysis, sixth on SWE-rebench) and Claude Opus 4.6's trajectory (fourth to first on SWE-rebench, but only fifth on Artificial Analysis) suggests they are sampling different problem distributions or applying different evaluation criteria rather than simply ranking the same capability differently.

Cole Brennan