The Inference Report

The SWE-rebench standings remain unchanged across the top tier, with AnthropicFable 5 holding 64.5%±1.41%, followed by GrokGrok 4.5 at 63.8%±0.60% and AnthropicOpus 5 at 63.4%±1.35%, indicating stability in the coding agent benchmark where the confidence intervals are tight enough to suggest genuine performance separation. On Artificial Analysis, one new entry appears: DeepSeek V4.1 Flash 1017 enters at rank 17 with 45.1, pushing prior entries down by one position, though the score falls within the established distribution and represents incremental rather than disruptive movement. The broader Artificial Analysis leaderboard shows typical churn in the 200-400 range where models cluster within narrow score bands (5-15 points), making positional shifts frequent but often meaningless; notably, the top 16 models remain identical to the prior snapshot, with Claude Opus 5.5 at 57.6 and Claude Sonnet 5.5 at 56 unchanged. SWE-rebench's methodology, evaluating code agents on real GitHub issues with structured evaluation, appears more discriminating than Artificial Analysis's broader assessment, as evidenced by the larger gaps between top performers (Fable 5 to Sonnet 5: 7.7 percentage points) versus the compressed middle sections of the general benchmark. The absence of movement in SWE-rebench's top 17 suggests either that these models have reached a performance plateau on this specific task or that the benchmark's variance estimates are sufficiently large to mask genuine improvements; the ±1.41% confidence interval around Fable 5's score, for instance, creates a band where competitors would need substantial gains to register as clear winners. Neither benchmark shows evidence of a category-defining shift, the data reflects incremental optimization within established model families rather than new architectural approaches breaking through.

Cole Brennan