The Inference Report

Claude Opus 4.6 holds the SWE-rebench top position at 65.3%, up from 53 on Artificial Analysis, a 12.3-point gain that reflects measurable improvement in solving real repository issues. The benchmark itself shows tighter clustering in the elite tier: gpt-5.2-2025-12-11-medium (64.4%), GLM-5 (62.8%), and Junie (62.8%) occupy positions 2 through 4, with only 1.6 percentage points separating them, whereas previously these models showed more dispersed rankings. Gemini 3.1 Pro Preview dropped from position 3 on Artificial Analysis to position 7 on SWE-rebench (57.2 to 62.3%), a counterintuitive movement that suggests the benchmarks measure different failure modes: Artificial Analysis may weight certain prompt-following or reasoning tasks more heavily, while SWE-rebench's repository-level problem-solving involves integrating code across files, managing context, and validating against actual test suites. The larger reorganization appears in the mid-tier: GLM-5 jumped from position 17 to position 3, Kimi K2.5 climbed from 29 to 16, and GLM-4.7 advanced from 44 to 14, gains of 14, 13, and 30 positions respectively, indicating these models have specific strengths in multi-step code repair that Artificial Analysis does not capture as prominently. The divergence between the two benchmarks is methodological: SWE-rebench tests models on unmodified real pull requests and requires executable validation, whereas Artificial Analysis appears to use a broader evaluation framework that may include instruction-following or synthetic tasks. Within SWE-rebench alone, the top 10 shows no movement, suggesting the highest-performing models have plateaued relative to each other, while the reshuffling below position 15 reflects genuine capability differences rather than noise.

Cole Brennan