The Inference Report

The SWE-rebench leaderboard shows stability at the top tier, with no movement in the top ten positions. AnthropicFable 5 maintains 64.5% ± 1.41%, followed by GrokGrok 4.5 at 63.8% ± 0.60% and AnthropicOpus 5 at 63.4% ± 1.35%, margins well within their confidence intervals. The Artificial Analysis benchmark, by contrast, reveals substantial reshuffling across the middle and lower ranks, with Claude Opus 5 rising to #1 at 63.1 (from #2 at 63.1), Claude Fable 5 at #2 with 62.1, and notable drops for models like DeepSeek V4 Pro, which fell from #16 at 53.2 to #30 at 45.3. The discrepancy between these two benchmarks warrants scrutiny: SWE-rebench uses controlled problem-solving conditions with uncertainty quantification, while Artificial Analysis provides point scores without error bounds, making direct comparison difficult. Within SWE-rebench's narrower scope, the consistency suggests the benchmark has sufficient resolution to distinguish performance in the 40-65% range but may be reaching saturation at the frontier, where confidence intervals begin to overlap meaningfully. The Artificial Analysis shifts hint either at different evaluation methodologies, data drift, or sensitivity to model versioning that SWE-rebench does not capture. Until both benchmarks publish their evaluation protocols in detail, the divergence remains interpretable but not fully explainable.

Cole Brennan