The Inference Report

The SWE-rebench rankings show no movement from the previous cycle, with AnthropicFable 5 holding the top position at 64.5% (±1.41%), followed by GrokGrok 4.5 at 63.8% (±0.60%) and AnthropicOpus 5 at 63.4% (±1.35%). The confidence intervals are wide enough that several adjacent pairs could plausibly swap on different test runs, yet the top tier remains stable across both measurement periods. By contrast, the Artificial Analysis benchmark presents a different ordering entirely, with Claude Opus 5 ranking first at 60.7 rather than second, and Claude Fable 5 at 59.9 rather than first, suggesting the two benchmarks are measuring distinct performance dimensions or using fundamentally different evaluation protocols. The SWE-rebench methodology appears to isolate specific coding task behaviors that differ from the broader capability assessment in Artificial Analysis, but without documentation of task construction, test set composition, or inter-rater reliability for either benchmark, it remains unclear whether the divergence reflects genuine performance variation across problem types or systematic differences in how each benchmark weights model capabilities. The lack of movement in SWE-rebench rankings over this period could indicate either stable model performance on this specific task distribution or insufficient statistical power to detect real changes given the uncertainty bands, particularly for models in the 40-60% range where error margins approach or exceed the gaps between adjacent entries.

Cole Brennan