The Inference Report

Claude Opus 4.6 holds the SWE-rebench top position at 65.3%, unchanged from prior measurement, while the Artificial Analysis benchmark shows material reshuffling across the field without shifts at the apex. On SWE-rebench, the top tier remains densely clustered between 60 and 65 percent, with gpt-5.2-2025-12-11-medium at 64.4% and GLM-5 and gpt-5.4-2026-03-05-medium both at 62.8%, suggesting diminishing returns in the high-performance region. Gemini 3.1 Pro Preview, ranked fifth on SWE-rebench at 62.3%, dropped from second on Artificial Analysis (57.2) but the discrepancy itself warrants scrutiny: the two benchmarks measure different problem domains and evaluation conditions, so direct ranking comparisons across them carry limited meaning. Notable climbers on Artificial Analysis include Kimi K2 Thinking, which jumped from position 37 (40.9) to position 17 (57.4), and Kimi K2.5, moving from 16 (46.8) to 13 (58.5), both suggesting Kimi's reasoning variants now handle the Artificial Analysis task distribution more effectively. The SWE-rebench methodology itself remains opaque in the provided data: without details on how tasks are selected, whether they stress particular failure modes, or how the evaluation handles partial credit, the stability of top rankings could reflect either genuine performance plateaus or ceiling effects in the benchmark design. The Artificial Analysis list's expansion to 340 entries and reordering throughout suggests either new model submissions or recalibration, but the data does not clarify which. Meaningful movement exists in the middle ranks on both benchmarks, but the absence of methodological documentation limits interpretation of whether these shifts reflect true capability divergence or measurement artifacts.

Cole Brennan