The Inference Report

The SWE-rebench rankings show no movement in the top tier, with Anthropic's Fable 5 holding at 64.5%, Grok 4.5 at 63.8%, and Opus 5 at 63.4%, all within their confidence intervals and unchanged from the previous cycle. The Artificial Analysis benchmark, by contrast, shows substantial reshuffling: Nemotron 3 Ultra 550B A55B enters at position 128, pushing Gemini 2.5 Pro Preview down one slot, a cascade that ripples through 297 positions below. Claude Opus 5 leads Artificial Analysis at 63.1, a 0.8-point gap from Fable 5 at 62.1, reversing the SWE-rebench ordering and suggesting the benchmarks measure different problem spaces or that the evaluation protocols diverge in their sensitivity to model capability. The SWE-rebench data carries tighter confidence bounds (Grok 4.5 at 0.60%, Junie Agent at 0.54%) compared to the broader spread in Artificial Analysis, indicating more controlled experimental conditions for the coding task, though the lack of movement across 17 entries raises questions about whether the test set has reached saturation or whether the evaluation is insensitive to recent model improvements. Artificial Analysis's dense 424-entry ranking with models clustered in the 1.0 to 3.7 range at the bottom suggests either a different difficulty calibration or inclusion of older or smaller-parameter models that SWE-rebench does not test, making direct comparison between the two benchmarks unreliable for assessing true performance shifts.

Cole Brennan