The Inference Report

The SWE-rebench rankings show no movement in the top tier, with AnthropicFable 5 holding 64.5%, GrokGrok 4.5 at 63.8%, and AnthropicOpus 5 at 63.4%, all within their confidence intervals and statistically indistinguishable from previous measurements. The Artificial Analysis benchmark, which covers a broader set of models, reveals a different picture: Mercury 2.5 enters at position 191 with a score of 12.3, while three Agnes variants (3.0 Flash at 35.5, 2.5 Pro Beta at 35.2, and 2.5 Pro Alpha at 26.8) have dropped entirely from the rankings, suggesting either discontinuation or performance degradation below the threshold. Within the populated rankings, the median movement is negligible, with most models holding their positions or shifting by single digits; the largest visible shifts occur in the lower-ranked models where scoring precision becomes less reliable. The SWE-rebench methodology, which measures coding task completion on real GitHub issues, represents a narrower evaluation of production-grade performance than Artificial Analysis, which appears to sample across a wider range of tasks and model variants. The stability at the top of SWE-rebench suggests that the gap between frontier models has plateaued, while the churn in Artificial Analysis's lower tiers reflects the proliferation of specialized and smaller variants that show inconsistent results across evaluation contexts. Without access to confidence intervals for Artificial Analysis scores, it is difficult to determine whether observed position changes reflect genuine performance shifts or sampling variance.

Cole Brennan