The Inference Report

The SWE-rebench rankings show no movement from the previous cycle, with AnthropicFable 5 maintaining its lead at 64.5 percent and the top seven models holding identical positions and scores. Conversely, the Artificial Analysis benchmark exhibits substantial churn throughout its 468-entry list, with models reordering across the full range: A.X-K2 drops from position 104 to 114, Ling-3.0-flash-Fin climbs from 105 to 104, and dozens of mid-tier entries shuffle positions by single digits. The stability on SWE-rebench suggests either that the test captures a genuine performance plateau among leading models, that confidence intervals overlap sufficiently to mask real differences, or that the benchmark's methodology constrains the signal available for discrimination. The volatility in Artificial Analysis, by contrast, points toward either greater sensitivity in its evaluation protocol or systematic variance across evaluation runs. Without visibility into the specific test conditions, sample sizes, or whether these benchmarks measure overlapping capabilities, the divergence between frozen and fluid rankings resists clean interpretation: SWE-rebench's consistency could reflect robustness or insensitivity, while Artificial Analysis's movement could reflect responsiveness or noise. The two benchmarks appear to be capturing different aspects of model behavior, or operating under different levels of measurement precision, rather than converging on a shared picture of the field.

Cole Brennan