The Inference Report

The SWE-rebench rankings remain static across the top tier, with AnthropicFable 5 holding 64.5% and the next five models clustered between 63.8% and 62.3%, confidence intervals tight enough that real separation is uncertain. The Artificial Analysis benchmark, by contrast, shows continuous churn: two new entries (Ling 3.1 Flash 1019 at rank 20 and Solar Pro 4 at rank 79) have displaced older models, and the full list has shifted down one position across most of the ranking. This pattern of new models entering while older ones fall suggests the Artificial Analysis benchmark is either refreshing its test set, adjusting its evaluation methodology, or capturing genuine performance improvements in recently released systems, whereas SWE-rebench appears to have reached a plateau where the top performers are stable. The SWE-rebench evaluation relies on resolving real software engineering tasks with defined success criteria, which tends to produce sticky rankings once a ceiling is approached; Artificial Analysis, measuring broader capability through what appears to be a wider or evolving set of tasks, shows more fluidity. Neither benchmark alone tells the full story: SWE-rebench's stability suggests the frontier on coding tasks is not moving rapidly, while Artificial Analysis's motion suggests either methodological drift or that general reasoning and instruction-following capabilities are still improving faster than specialized code-solving ability.

Cole Brennan