The Inference Report

The SWE-rebench rankings remain frozen while the Artificial Analysis leaderboard shuffled 38 entries, introducing Mistral Large 4 Preview at position 32 and cascading everything below it downward. On SWE-rebench, AnthropicFable 5 holds 64.5% with tight confidence intervals across the top tier, and the spread between first and tenth place spans 12.8 percentage points, suggesting real separation in coding task performance. The Artificial Analysis benchmark, by contrast, shows a different stratification: Claude Opus 5.5 leads at 57.6, but the top ten compress within a 5-point band, and models ranked 250 onward cluster between 9 and 5 percent with minimal differentiation. The methodology concern here is acute. SWE-rebench appears to test concrete problem-solving on real repository issues with controlled evaluation, whereas Artificial Analysis likely aggregates diverse metrics across broader capability domains where coding may be one signal among many. The two benchmarks rank models differently enough to suggest they measure distinct things: SWE-rebench's top performer (Fable 5 at 64.5%) ranks third on Artificial Analysis (53.4), while Artificial Analysis's leader (Opus 5.5 at 57.6) sits ninth on SWE-rebench (56.8%). This divergence is not noise. It reflects whether a benchmark prioritizes narrow coding competence or distributed capability. Neither ranking change signals meaningful progress without knowing whether the Mistral insertion reflects actual improvement or simply expanded roster coverage, and the SWE-rebench stability suggests either the evaluation is mature or updates are infrequent.

Cole Brennan