The Inference Report

The SWE-rebench leaderboard holds stable at the top, with AnthropicFable 5 maintaining 64.5% and the top five models clustered within 2.2 percentage points, confidence intervals overlapping substantially enough that ranking shifts between them would not represent genuine performance differences. The Artificial Analysis benchmark, by contrast, shows considerable churn across its 422-entry list, with models shuffling positions frequently even when scores differ by 0.1 to 0.3 points, a pattern that raises questions about whether such fine-grained distinctions reflect real capability gaps or measurement noise. Notable movements include Ling 3.0 Flash dropping from rank 59 (37.8) to unranked status, NVIDIA Nemotron 3 Super 120B falling from 119 (25.7) to unranked, Ling 3.0 Tiny climbing from 125 (23.9) to 121 (24.5), and Phi-4 Mini Instruct rising from 326 (5.7) to 316 (6.2). The SWE-rebench methodology, which tests agents on actual software engineering tasks with controlled evaluation conditions, produces the more interpretable signal: the top tier of models genuinely solves 56 to 65 percent of problems, with uncertainty bands that reflect real variability. The Artificial Analysis scores, spanning from 64.5 down to 1.0 across hundreds of entries, compress evaluation into a single dimension that may conflate different failure modes or reflect task-specific brittleness rather than general capability. Without visibility into Artificial Analysis's evaluation protocol, whether it uses held-out test sets, how it handles edge cases, or whether scoring is normalized, the frequent micro-movements and aggressive differentiation at the tail end suggest the ranking may be sensitive to evaluation artifacts rather than tracking reproducible differences in model behavior.

Cole Brennan