The Inference Report

The SWE-rebench standings show no movement at the top tier, with AnthropicFable 5 holding 64.5% and the next four positions (Grok 4.5, Opus 5, GLM-5.2, GPT-5.6 Sol) maintaining their rankings between 62.3% and 63.8%, all within margins that overlap their confidence intervals. The Artificial Analysis benchmark, by contrast, reveals substantial reordering across its full 422-entry leaderboard: Claude Opus 5 climbed to first place with 63.1 (up 2.4 points from 60.7), displacing Fable 5 to second at 62.1, while GPT-5.6 Sol moved from third to third with a 2.0-point gain to 60.9, and deeper in the list models like Muse Spark 1.2 jumped nine positions and 2.7 points to number 7. The divergence between these two benchmarks reflects their different methodologies: SWE-rebench measures coding agent performance on concrete repository tasks with tight confidence intervals that constrain apparent movement, while Artificial Analysis appears to evaluate broader model capabilities with larger score spreads. The top-tier SWE-rebench stability suggests either that the models tested are genuinely separated by meaningful gaps (the 1.1-point spread between first and fifth is real), or that the benchmark's variance is high enough to mask smaller changes, whereas Artificial Analysis's volatility in the 30-50 rank range and consistent 1-3 point gains across the board hints at either a recalibrated evaluation, a different test set, or systematic score inflation. Without knowing whether Artificial Analysis changed its methodology or simply re-ran the same test, the pattern is ambiguous: the gains could reflect genuine model improvement, measurement drift, or both.

Cole Brennan