The Inference Report

The SWE-rebench rankings remain static across both snapshots, with AnthropicFable 5 holding 64.5% and the top seventeen models unchanged in position and score. This stability itself is noteworthy: the coding benchmark shows no movement whatsoever, suggesting either that evaluation cycles are infrequent or that the models assessed have not undergone meaningful revision. The confidence intervals are tight enough to rule out measurement noise as an explanation for the lack of change (GrokGrok 4.5's ±0.60% is especially precise), indicating the benchmark has sufficient resolution to detect real differences. By contrast, the Artificial Analysis rankings show extensive reshuffling throughout the list, with Step 5 Preview entering at #11, GPT-5.6 Terra dropping from #11 to #12, and cascading shifts down the entire leaderboard. This divergence between benchmarks raises a methodological question: SWE-rebench appears to measure something more stable or resistant to model iteration than the Artificial Analysis suite, or the two are sampling fundamentally different capability dimensions. Without knowing the composition or evaluation protocol of either benchmark, it is difficult to assess whether the SWE-rebench stasis reflects genuine plateau in coding agent performance or simply less frequent or less sensitive evaluation. The Artificial Analysis churn, conversely, could indicate either rapid iteration in the broader model ecosystem or greater sensitivity to minor capability shifts, but the absence of clear winners and losers across the full 459-model ranking makes it hard to extract signal about what has actually improved.

Cole Brennan