The Inference Report

Claude Opus 4.6 holds the SWE-rebench lead at 65.3%, unchanged from the previous cycle, while the tier immediately below shows modest compression: gpt-5.2-2025-12-11-medium sits at 64.4%, and GLM-5 and gpt-5.4-2026-03-05-medium both score 62.8%. The meaningful movement occurs in the mid-field, where GLM-4.7 has climbed from rank 43 (42.1 points on Artificial Analysis) to rank 14 (58.7% on SWE-rebench), a shift that suggests either a genuine capability jump or a divergence in what these two benchmarks measure. Kimi K2.5 similarly advanced from rank 28 to rank 16, and Kimi K2 Thinking jumped from rank 53 to rank 21, indicating Chinese models have made gains on the SWE-rebench evaluation specifically. Gemini 3.1 Pro Preview dropped from rank 3 to rank 6 on SWE-rebench (62.3%) despite holding rank 3 on Artificial Analysis (57.2), a discrepancy that raises questions about benchmark stability or whether SWE-rebench and Artificial Analysis weight different problem classes. The Artificial Analysis leaderboard itself shows minimal reshuffling in the top 20, with GPT-5.5 leading at 60.2 and Claude Opus 4.7 at 57.3, suggesting those rankings have stabilized. At the lower end, Granite 4.1 models appear as new entries on Artificial Analysis (30B at rank 229, 8B at 261, 3B at 324), and QwQ 32B and Qwen3 VL 30B A3B swapped positions at ranks 160 and 161 without score change, a cosmetic reordering. The lack of dramatic score inflation across either benchmark and the persistence of the same top performers suggest the evaluations are not drifting, though the divergence between SWE-rebench and Artificial Analysis rankings for mid-tier models warrants investigation into whether they stress different failure modes or simply employ different evaluation protocols.

Cole Brennan