The Inference Report

The SWE-rebench rankings remain static across the top tier, with AnthropicFable 5 holding 64.5% ± 1.41% at first place and the next six positions unchanged through OpenAICodexAgent at 58.0% ± 1.29%. This stability in the coding benchmark suggests either that the test set has reached saturation at the frontier or that recent model updates have not substantially altered performance on this particular evaluation. The Artificial Analysis benchmark, by contrast, shows considerable churn: Claude Sonnet 5.5 enters at rank 2 with 56.0, displacing Claude Fable 5.1 down one position to 3, while the rest of the top 20 shifts accordingly with no score changes, only ranking adjustments. The gap between the two benchmarks is worth noting. SWE-rebench scores cluster tightly in the 60s and 50s for leading models, with error bars ranging from ±0.54% to ±1.83%, reflecting controlled experimental conditions. Artificial Analysis scores begin at 57.6 for Claude Opus 5.5 and decline more steeply down the list, reaching single digits by rank 250. This divergence hints at different task difficulty profiles: SWE-rebench may be measuring a narrower, more saturated capability space (software engineering problem solving with defined test cases), while Artificial Analysis appears to cover a broader skill distribution. Neither benchmark reveals methodological details in the provided data, making it difficult to assess whether the stability in SWE-rebench reflects genuine parity or simply coarse-grained measurement. The entry of Claude Sonnet 5.5 into Artificial Analysis's top two is the only concrete movement worth tracking, but without prior scores for this model on that benchmark, its significance cannot be determined.

Cole Brennan