The Inference Report

The SWE-rebench rankings show no movement from the previous cycle: the top twelve entries remain in identical positions with matching scores and confidence intervals. AnthropicFable 5 holds first at 64.5% ± 1.41%, followed by GrokGrok 4.5 at 63.8% ± 0.60% and AnthropicOpus 5 at 63.4% ± 1.35%, with all downstream positions locked through rank seventeen. The stability across a full benchmark cycle suggests either that the test set has reached saturation or that the underlying model capabilities have plateaued relative to the task difficulty. On Artificial Analysis, by contrast, substantial reshuffling occurred in the top tier: Muse Spark 1.3 and GPT-6 Astra entered the ranking as new entries at positions three and five respectively, pushing prior occupants down, while Claude Fable 5.1 moved to first place at 65.7 from a prior position lower in the list. The broader Artificial Analysis leaderboard shows consistent but minor position shifts throughout the middle ranks, with no entries dropping off entirely and only two confirmed new additions in the visible window. The divergence between the two benchmarks, perfect stability on SWE-rebench versus active reordering on Artificial Analysis, raises a methodological question: SWE-rebench's narrow confidence intervals and lack of movement across its top entries suggest either tighter experimental controls or a ceiling effect that Artificial Analysis does not exhibit. Without access to sample sizes or retest protocols for either benchmark, it remains unclear whether the SWE-rebench plateau reflects genuine convergence in model performance or methodological constraints that limit discriminative power at the frontier.

Cole Brennan