The Inference Report

The SWE-rebench rankings show no movement from the previous cycle, with AnthropicFable 5 holding 64.5% ± 1.41% at the top, followed by GrokGrok 4.5 at 63.8% ± 0.60% and AnthropicOpus 5 at 63.4% ± 1.35%, but the Artificial Analysis benchmark reveals substantial churn across its 421-model roster, with two new entries disrupting the upper tier: Qwen3.8 Max enters at number 5 with 56.2, pushing Claude Opus 4.8 down one slot to 6, and Muse Spark 1.2 debuts at number 9 with 54.1, while Ling-3.0-flash appears at number 57 with 37.4. The stability in SWE-rebench contrasts sharply with the Artificial Analysis churn and raises a methodological question: SWE-rebench uses confidence intervals (ranging from ± 0.54% to ± 1.83%) suggesting repeated trials or statistical sampling, whereas Artificial Analysis reports single point scores without uncertainty estimates, making it unclear whether the apparent volatility reflects genuine model improvements, evaluation methodology shifts, or simply different statistical rigor between benchmarks. The top-tier SWE-rebench models cluster tightly between 62.3% and 64.5%, overlapping within their confidence bands, which means the ranking order itself carries limited discriminative power at that level. Without historical Artificial Analysis rankings to confirm whether these entries are genuinely new models or data collection artifacts, the apparent movement cannot be cleanly separated from benchmark drift.

Cole Brennan