The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, a 12.3-point jump from its previous ranking of 53 on Artificial Analysis, while Gemini 3.1 Pro Preview has dropped from first place (57.2 on Artificial Analysis) to fifth on SWE-rebench at 62.3%, suggesting the two benchmarks measure different aspects of code generation or that SWE-rebench's evaluation methodology surfaces different model capabilities than Artificial Analysis does. GLM-5 and Kimi K2.5 both show substantial gains on SWE-rebench relative to their Artificial Analysis scores, rising to third (62.8% versus 49.8%) and thirteenth (58.5% versus 46.8%) respectively, while several models near the top of SWE-rebench appear lower on Artificial Analysis, indicating either a shift in what tasks dominate the coding benchmark or differences in how the two evaluations weight problem difficulty and solution quality. The SWE-rebench methodology itself remains opaque from the data provided: the scoring scale differs from Artificial Analysis, the test set composition is unspecified, and whether improvements reflect genuine capability gains or benchmark-specific optimization cannot be determined from ranking movement alone. What is clear is that SWE-rebench produces a substantially different ordering among frontier models, which matters if teams are using it to guide development priorities, but without documentation of the benchmark's task distribution, evaluation harness, and baseline stability, the practical significance of these shifts remains ambiguous.

Cole Brennan