The Inference Report

The SWE-rebench rankings show no movement at the top tier, with Claude Code, Junie, and Claude Opus 4.6 holding their positions at 52.9%, 52.1%, and 51.7% respectively, while the Artificial Analysis benchmark reveals more volatility across its 323-model leaderboard. Claude Opus 4.5 dropped significantly on SWE-rebench from a reported 49.7 on Artificial Analysis to 43.8%, falling from position 8 to 12, a 5.9-point decline that suggests either a methodology shift between benchmarks or performance degradation under the SWE-rebench evaluation protocol. Kimi K2 Thinking climbed 15 positions on Artificial Analysis (from 40.9 to 43.8 on SWE-rebench, now ranking 13th), while GLM-5 fell 7 positions despite maintaining reasonable scores, dropping from 49.8 on Artificial Analysis to 42.1 on SWE-rebench. The divergence between Gemini 3 Pro Preview's performance on the two benchmarks is notable: it scores 48.4 on Artificial Analysis but only 46.7 on SWE-rebench, ranking 11th versus 8th respectively, suggesting the benchmarks weight different aspects of coding capability or test different problem distributions. Mid-tier models show the most churn: Kimi K2.5 dropped 8 positions on SWE-rebench (37.9%) despite scoring 46.8 on Artificial Analysis, and GLM-4.6 climbed from position 53 to 22 on Artificial Analysis but remains at 37.1% on SWE-rebench at position 22, indicating the benchmarks are not measuring identical capabilities. The absence of clear correlation between SWE-rebench and Artificial Analysis rankings across the full dataset suggests these evaluations are testing distinct problem spaces or that the SWE-rebench protocol imposes constraints (likely around repository-level code generation and integration) that don't map cleanly to general coding performance.

Cole Brennan