The Inference Report

The SWE-rebench standings show minimal movement at the top, with AnthropicFable 5 holding the lead at 64.5% (±1.41%) and the top six models clustered within 2.5 percentage points, but the Artificial Analysis benchmark reveals a sharp, systematic decline across the entire leaderboard that defies the stability suggested by SWE-rebench alone. Claude Fable 5.1 dropped from 65.7 to 56.8, a 13% relative decline, while GPT-6 Astra fell from 61.2 to 54.7, and Claude Opus 5 from 63.1 to 54.1, suggesting either a methodological shift in Artificial Analysis's evaluation protocol or a recalibration of its test set that hit all models uniformly rather than differentially. The pattern is not random: every model in the Artificial Analysis top 100 lost between 8 and 15 percentage points, with the median drop around 11 points, while lower-ranked models (those below rank 200) showed smaller absolute losses, implying the benchmark either tightened its criteria, introduced harder test cases, or corrected for prior inflation. The SWE-rebench data, by contrast, appears internally consistent with its previous iteration, suggesting these are genuinely separate evaluation regimes measuring different aspects of code generation capability. Without access to Artificial Analysis's methodology change documentation, the most parsimonious explanation is that one benchmark recalibrated while the other did not, making cross-benchmark comparisons unreliable and highlighting the risk of relying on a single leaderboard source for model assessment.

Cole Brennan