The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, up from fourth place and 53% on the prior Artificial Analysis benchmark, representing a 12.3-point gain that reflects either improved model capability or a meaningful shift in how the two benchmarks weight code generation tasks. The top five models cluster tightly between 62.3% and 65.3% on SWE-rebench, with gpt-5.2-2025-12-11-medium at 64.4%, GLM-5 and gpt-5.4-2026-03-05-medium both at 62.8%, and Gemini 3.1 Pro Preview at 62.3%, all of which moved up relative to their prior rankings. Kimi K2.5 climbed from 16th place (46.8 points) to 13th (58.5%), while Kimi K2 Thinking advanced from 36th (40.9 points) to 17th (57.4%), suggesting that reasoning-focused variants are closing gaps on general-purpose models in software engineering tasks. The two benchmarks diverge materially in their orderings: gpt-5.4 ranks first on Artificial Analysis at 57.2 but fourth on SWE-rebench at 62.8, while Gemini 3.1 Pro Preview ties at first on Artificial Analysis but ranks fifth on SWE-rebench, implying that SWE-rebench either captures different failure modes in code generation or applies stricter evaluation criteria around execution correctness rather than response quality alone. The spread between top and middle performers narrowed on SWE-rebench (Claude Opus to Step-3.5-Flash spans 5.7 percentage points) compared to Artificial Analysis (GPT-5.4 to MiMo-V2-Pro spans 8 points), which could indicate that SWE-rebench has tighter clustering due to smaller sample sizes or more binary pass-fail scoring, though neither benchmark publication provides explicit methodology details sufficient to confirm the source of the divergence.

Cole Brennan