The Inference Report

The SWE-rebench rankings remain static across both snapshots, with AnthropicFable 5 holding first place at 64.5% and the top seventeen models unchanged in position and score. The Artificial Analysis benchmark, by contrast, shows wholesale reshuffling: Grok 4.7 and MiMo-V2.6-Pro enter the top ten as new entries at positions 7 and 8, displacing Qwen3.8 Max and GLM-5.3 downward by two spots each, while every other model in the 300-entry list shifts position despite most scores remaining identical to the previous cycle. This pattern suggests the Artificial Analysis ranking operates on a different methodology than SWE-rebench, likely incorporating recency weighting, model release date, or other temporal factors that cause constant reordering without corresponding score changes, whereas SWE-rebench appears to measure against a fixed problem set with genuine performance stability at the top. The stability of SWE-rebench scores across the full range, paired with tight confidence intervals (most under 1.5%), indicates controlled experimental conditions and reproducible results; Artificial Analysis's perpetual repositioning of identical-scoring models raises questions about whether the ranking reflects meaningful differentiation or administrative sorting unmoored from the underlying evaluation data. Without visibility into how Artificial Analysis recalculates rankings when scores don't change, treating its movement as a signal of capability shift would be premature.

Cole Brennan