The Inference Report

The SWE-rebench rankings remain static at the top tier, with AnthropicFable 5 holding 64.5% and the next five models clustered within 2.2 percentage points, all showing confidence intervals that overlap substantially with their neighbors. Artificial Analysis data, by contrast, reveals significant churn across its 470-entry leaderboard, particularly in the middle ranks where Granite 4.2 30B has shifted from position 163 to 193, and several DeepSeek variants have moved multiple positions, though the magnitude of these shifts often falls within typical measurement noise for scores in the 12-15 range. The two benchmarks measure different problem spaces: SWE-rebench evaluates real-world software engineering tasks with controlled conditions and reported error margins, while Artificial Analysis appears to sample a broader cross-section of models with no disclosed methodology or uncertainty quantification. At the SWE-rebench top, the stability reflects either genuine convergence in capability or plateau in discrimination power; the absence of new entries in the top ten positions suggests the frontier has settled. Artificial Analysis's fluidity at ranks 160-200 warrants skepticism about whether those shifts represent meaningful performance changes or noise from unmeasured experimental variance. Neither benchmark clarifies whether the apparent separation between Fable 5 (64.5%) and the field reflects a genuine algorithmic advance or saturation effects on the task distribution itself.

Cole Brennan