The Inference Report

The SWE-rebench coding benchmark shows no movement at the top tier, with AnthropicFable 5 holding 64.5 percent, GrokGrok 4.5 at 63.8 percent, and AnthropicOpus 5 at 63.4 percent, their positions and confidence intervals unchanged from the previous cycle. The Artificial Analysis benchmark, by contrast, exhibits substantial churn across its 473-model ranking, though the shifts are almost entirely cosmetic: most models moved one to three positions while their scores remained identical to prior measurements, suggesting the list was simply resorted or expanded with new entries rather than re-evaluated. For example, MiMo-V2.5 dropped from position 86 to 116 despite maintaining a 25.2 score, while Mercury 2 fell from 177 to 228 with no score change, and Llama 4 Maverick moved from 235 to 253 at 10.0 points. A small number of models did show genuine score shifts: Qwen3 32B improved from 8.6 to 7.3, Qwen3 14B fell from 8.2 to 7.6, and Mistral Small 3.2 dropped from 8.2 to 7.0, though the direction and magnitude of these changes lack clear pattern. The methodology of Artificial Analysis remains opaque, with no published evaluation protocol or reproducibility details, making it impossible to assess whether score changes reflect model improvements, benchmark refinement, or measurement noise. The SWE-rebench data, by contrast, includes confidence intervals that permit statistical interpretation, though the absence of any top-tier movement over this period suggests either genuine performance saturation at the frontier or insufficient sensitivity in the benchmark to detect improvements when they occur.

Cole Brennan