The Inference Report

The SWE-rebench rankings hold steady at the top, with AnthropicFable 5 maintaining 64.5% and the next four models unchanged in position and score, suggesting the coding agent benchmark has stabilized around these performers. On the Artificial Analysis evaluation, the movement is modest but directional: GPT-6 Astra slipped from 52.8 to 52.7, Claude Fable 5 dropped from 49.7 to 49.6, and most models in the upper tiers lost 0.1 to 0.3 points, indicating either marginal measurement variance or a tightening of performance gaps as the field matures. Lower-ranked models show more volatility, with Inkling Small jumping from 26.1 to 27.8 (rank 69 to 57), Ling 3.0 Tiny climbing from 11.9 to 15.3 (rank 194 to 151), and Gemma 4 31B surging from 15.4 to 19.0 (rank 149 to 125), suggesting either improved model versions or recalibration of the evaluation methodology for smaller models. The Artificial Analysis benchmark's broader coverage (459 models versus 17 on SWE-rebench) means its shifts reflect both genuine capability changes and the inherent noise of a more distributed leaderboard, where single-point movements can swing rankings significantly. Without access to confidence intervals on the Artificial Analysis scores, it is difficult to distinguish signal from noise in the middle and lower tiers, and the modest fractional changes at the top suggest the benchmark lacks sufficient granularity to detect meaningful progress in the current generation of frontier models.

Cole Brennan