The Inference Report

The SWE-rebench rankings show no movement from the previous report: the top tier remains unchanged, with GPT-5.5-xhigh at 62.7%, JunieAgent at 61.6%, and CodexAgent at 60.4%, each holding their positions across the full 24-model list. The Artificial Analysis benchmark, by contrast, exhibits substantial churn across its 398-entry ranking, though the top tier again proves stable, Claude Fable 5 holds the lead at 59.9, followed by Claude Opus 4.8 at 55.7 and GPT-5.5 at 54.8. Below that summit, however, the ordering has shifted measurably: GPT-5 mini dropped from #65 at 33.0 to #72 at 30.9, a loss of 2.1 points and seven positions; Mistral Small 4 fell from #126 at 20.8 to #132 at 19.6; and Qwen3 Next 80B A3B plummeted from #134 at 19.8 to #159 at 16.7, suggesting either methodological revision or genuine performance variance in the 16-20 point band where many models cluster. The SWE-rebench's immobility raises a question about whether those agentic benchmarks are less sensitive to model updates than Artificial Analysis, or whether the coding agents themselves have stabilized while the underlying base models continue to diverge. The Artificial Analysis instability in the mid-range, where confidence intervals would overlap, warrants scrutiny of whether those score differences exceed measurement error; without published confidence bounds for that benchmark, the ranking shifts read as plausible but not necessarily meaningful.

Cole Brennan