The Inference Report

The SWE-rebench rankings show no movement from the previous cycle, with AnthropicFable 5 holding the top position at 64.5%±1.41%, followed by GrokGrok 4.5 at 63.8%±0.60% and AnthropicOpus 5 at 63.4%±1.35%. The coding benchmark remains stable across its top seventeen entries, suggesting either that the evaluation window was too short to detect meaningful shifts or that these models have reached a plateau on this particular task distribution. The confidence intervals are tight enough that small variations would be noise rather than progress. Artificial Analysis shows considerably more churn, with Agnes 2.5 Pro Beta entering at position 42 as a new entry while the rest of the leaderboard shuffled downward by one position each, a pattern consistent with a single insertion rather than wholesale reordering. The median score in the Artificial Analysis top 100 hovers around 20 points, roughly one-third the performance of top SWE-rebench models, indicating the two benchmarks measure different problem classes or that Artificial Analysis weights harder instances more heavily. Without knowing the methodology behind Artificial Analysis or whether its test set was refreshed, it is difficult to assess whether the insertion of Agnes 2.5 Pro Beta represents genuine capability advancement or dataset drift. The SWE-rebench stability is more trustworthy given its explicit error bars, but it also invites the question of whether the benchmark is sensitive enough to detect incremental improvements, or whether the top performers have simply saturated the task space.

Cole Brennan