The Inference Report

The SWE-rebench results show no movement in the top tier: AnthropicFable 5 holds position one at 64.5% with confidence intervals tight enough to establish clear separation from Grok 4.5 at 63.8% and Opus 5 at 63.4%. All three models maintain their prior rankings, and the spread between first and fifth place remains 2.2 percentage points, suggesting the frontier on coding tasks has plateaued or these measurements have reached the precision limits of the benchmark itself. The Artificial Analysis benchmark presents a different picture: Gemini 3.8 Flash enters at position eight with 58.7, displacing Qwen3.8 Max to ninth, while the rest of the top twenty shifts down one position. This insertion is methodologically notable because Artificial Analysis covers a broader evaluation surface than SWE-rebench alone, and the entry of a Google model in the upper ranks hints at different task distributions or evaluation protocols between the two datasets. However, the discrepancy between rankings raises a practical question: SWE-rebench measures direct code generation on real repository issues with verifiable outcomes, whereas Artificial Analysis appears to aggregate multiple capability dimensions. The coding-specific benchmark shows mature performance clustering at the top, with Anthropic and OpenAI variants dominating, while the broader benchmark permits more competition from models like Gemini that may excel on non-coding tasks. Without knowing whether Artificial Analysis includes or weights SWE-rebench results, the gap between these two leaderboards suggests they are measuring related but distinct problems, and neither alone captures the full picture of model capability on engineering tasks.

Cole Brennan