The Inference Report

The SWE-rebench leaderboard shows no movement at the top tier, with AnthropicFable 5, GrokGrok 4.5, and AnthropicOpus 5 holding positions 1 through 3 at 64.5%, 63.8%, and 63.4% respectively, all within their reported confidence intervals. The Artificial Analysis benchmark tells a different story: Gemini 4 Carbon gained 0.6 points to reach 60.0, Claude Opus 5.5 improved 0.7 points to 59.1, and two new entries appeared (GPT-6 Sol at position 11 with 44.2 and GPT-5.6 Terra at position 18 with 32.3), suggesting either model releases or evaluation methodology adjustments not reflected in the SWE-rebench data. The gap between the two benchmarks is instructive: SWE-rebench clusters the top five models within 2.2 percentage points, a tight range that reflects the difficulty of the task and the convergence of high-capacity models on this specific evaluation, while Artificial Analysis spreads the same performers across a 3.1 point range, indicating different test construction or scoring criteria. Within SWE-rebench, the confidence intervals are narrow enough that single-point movements would be meaningful, yet none occurred, suggesting either genuine stability in model performance or evaluation variance that masks real differences. The lower-ranked models show marginal gains across Artificial Analysis (most under 0.3 points), consistent with noise rather than systematic improvement. Neither benchmark provides evidence of a qualitative shift in coding capability this cycle.

Cole Brennan