The Inference Report

The SWE-rebench rankings remain static across the top tier, with AnthropicFable 5 holding 64.5% and the next four positions unchanged through GPT-5.6 Sol at 62.3%. The stability here reflects narrow confidence intervals (most under 1.8 percentage points), suggesting these measurements have settled into reliable territory. Artificial Analysis, by contrast, shows substantial churn below the top twenty entries, with models like LFM2.5-1.2B-Thinking newly appearing at rank 402 and numerous repositionings throughout the mid-tier that indicate either methodological differences between the two benchmarks or genuine performance variance on different evaluation sets. The gap between SWE-rebench's top performer and its seventeenth-ranked model spans 47.4 percentage points (64.5% to 17.1%), whereas Artificial Analysis compresses the same span from rank 1 to 17 into just 9.9 points (63.1 to 53.2), suggesting SWE-rebench may be testing a narrower or more discriminative problem space. Within Artificial Analysis, the long tail below rank 100 shows models clustering tightly at single-digit scores, making positional shifts there largely noise. The absence of movement in SWE-rebench's top ten over this cycle indicates either that the benchmark has reached saturation at its current scale or that frontier models are converging on its difficulty floor, a pattern worth examining against the benchmark's design rather than interpreting as stagnation in model capability.

Cole Brennan