The Inference Report

The SWE-rebench rankings show substantial churn at the top, though the movement warrants scrutiny. Two new GPT-5.5 variants (xhigh and medium) now occupy positions 1 and 4 at 62.7% and 58.9%, displacing previously dominant models, while Codex and Claude Code jumped from positions 18 and 17 to positions 2 and 3, gaining 2.1 and 1.2 percentage points respectively. Claude Opus 4.6 fell from first place (65.3%) to sixth (53.1%), a 12.2-point drop that demands explanation, and several formerly high-ranked models (gpt-5.2-2025-12-11-medium, Junie, DeepSeek-V3.2, Claude Sonnet 4.5, Qwen3.5-397B-A17B) disappeared entirely from the benchmark. The Artificial Analysis rankings remain largely stable with identical scores and positions, suggesting the volatility is specific to SWE-rebench's evaluation methodology or dataset. Without documentation of what changed in the benchmark itself, whether test cases were added, removed, or reweighted, or whether evaluation criteria shifted, it is unclear whether these movements reflect genuine capability differences or artifacts of the measurement apparatus. The scale of Claude Opus 4.6's decline particularly raises questions: such a large score regression without corresponding changes in the model itself points toward benchmark modifications rather than model degradation. Until the SWE-rebench evaluation protocol is transparently specified, these rankings indicate movement but not necessarily meaningful progress.

Cole Brennan