The Inference Report

Claude Opus 4.6 moved from fourth to first on SWE-rebench, climbing 12.3 percentage points from 53 to 65.3 percent, while Gemini 3.1 Pro Preview dropped from the top Artificial Analysis ranking at 57.2 to sixth on SWE-rebench at 62.3 percent. The gap between first and second place on SWE-rebench narrowed to just 0.9 points (Claude Opus 4.6 at 65.3 versus gpt-5.2-2025-12-11-medium at 64.4), and the top six models now cluster between 62.3 and 65.3 percent, suggesting convergence at the frontier rather than separation. GLM-5 and GLM-5.1 each gained roughly 13 points, moving from tenth and seventh on Artificial Analysis to third and fifth on SWE-rebench, indicating that coding-specific evaluation surfaces different capability profiles than the broader Artificial Analysis benchmark. However, the two benchmarks tell divergent stories about the field: SWE-rebench shows tight competition in the 58 to 65 percent range across the top twenty models, while Artificial Analysis exhibits steeper stratification, with the top performer at 57.2 and a sharper drop-off below rank fifty. The methodological difference matters here. SWE-rebench measures repository-level code generation against real GitHub issues with deterministic evaluation criteria, while Artificial Analysis covers broader reasoning and general capability. Models like Claude Opus 4.6 and the GLM family appear better calibrated to the specific constraints of software engineering tasks, but without visibility into whether SWE-rebench's evaluation methodology changed or whether these represent genuinely new model versions, the magnitude of the gains (particularly the 12-point jump for Claude) warrants scrutiny of whether the benchmark itself remained stable or whether the test set, evaluation harness, or scoring logic shifted.

Cole Brennan