The Inference Report

Claude Opus 4.6 maintains its position at the top of the SWE-rebench rankings with 65.3%, unchanged from the previous evaluation, while the tier beneath it shows marginal movement with gpt-5.2-2025-12-11-medium and GLM-5 holding positions two and three at 64.4% and 62.8% respectively. The most significant repositioning occurs in the middle ranks: Kimi K2.5 climbs from position 16 (46.8 on Artificial Analysis) to position 13 (58.5% on SWE-rebench), and Kimi K2 Thinking advances from position 34 (40.9) to position 17 (57.4%), suggesting these models have received targeted improvements in code-related task handling that the Artificial Analysis benchmark has not yet captured. The divergence between SWE-rebench and Artificial Analysis rankings is particularly notable at the top: Claude Opus 4.6 ranks first on SWE-rebench at 65.3% but only fourth on Artificial Analysis at 53, indicating that SWE-rebench measures a narrower, more specialized capability in software engineering tasks where Claude's advantage is pronounced, while Artificial Analysis distributes scores more evenly across a broader capability spectrum. Below the top tier, the rankings remain largely stable, with most models holding their relative positions, though the absolute gaps between SWE-rebench and Artificial Analysis scores widen considerably for lower-ranked models, suggesting that coding benchmarks and general capability benchmarks increasingly diverge as model sophistication decreases. The methodology distinction matters here: SWE-rebench appears to test repository-level engineering tasks with higher fidelity to real-world scenarios, while Artificial Analysis likely employs a different evaluation protocol, making direct score comparison problematic even when rankings partially align.

Cole Brennan