The Inference Report

On the SWE-rebench, the top tier remains static: OpenAI's gpt-5.5-2026-04-23-xhighModel holds position one at 62.7 percent, followed by JunieAgent at 61.6 percent and OpenAI's CodexAgent at 60.4 percent. The confidence intervals are tight enough to distinguish these leaders, with standard errors ranging from 0.53 to 1.98 percentage points across the ranked set, suggesting the evaluation captures consistent performance differences. Across the Artificial Analysis benchmark, the landscape shifts more dramatically: Claude Fable 5 now leads at 59.9, displacing GPT-5.5 from the top spot, while Claude Opus 4.8 sits second at 55.7 and GPT-5.5 drops to third at 54.8. The gap between the SWE-rebench's top performer and Artificial Analysis's top performer is 2.8 points, a meaningful divergence that hints at different problem structures. Within Artificial Analysis, the middle ranks show considerable churn: Llama 3.3 Instruct 70B climbed from position 258 to 242, a 16-rank jump, while several models in the 240 to 260 range shuffled positions, suggesting modest score movements in a crowded band where many models cluster between 8 and 10 points. The two benchmarks do not track perfectly: models strong on SWE-rebench (like JunieAgent and OpenAI's variants) do not appear on the Artificial Analysis list, and vice versa, indicating the tests measure distinct capabilities rather than a single underlying skill. The SWE-rebench concentrates on code-generation agents in controlled conditions, while Artificial Analysis appears broader and less transparent in methodology, making direct comparison hazardous. Without historical Artificial Analysis data from prior runs, the significance of Claude Fable 5's ascent to first cannot be evaluated; the SWE-rebench's stability suggests real differences in agent capability, but the Artificial Analysis movements may reflect noise or evaluation drift rather than genuine progress.

Cole Brennan