The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, up from 53 on the Artificial Analysis benchmark, while Gemini 3.1 Pro Preview dropped from first place (57.2 on Artificial Analysis) to fifth (62.3% on SWE-rebench), and Kimi K2.5 climbed from 46.8 to 58.5%, a gain of 11.7 percentage points. The SWE-rebench scores are substantially higher across the board than the Artificial Analysis scores for the same models, suggesting either a difference in task difficulty, evaluation methodology, or the benchmarks' sensitivity to specific coding patterns. GLM-5 moved from tenth place (49.8) to third (62.8%), and Kimi K2 Thinking jumped from 40.9 to 57.4%, indicating that certain architectures perform disproportionately better on the SWE-rebench evaluation. The clustering of models between 58 and 65 percent on SWE-rebench, compared to the wider spread on Artificial Analysis, raises questions about whether SWE-rebench's task distribution favors certain model families or whether its evaluation criteria reward specific coding strategies. Without explicit information about SWE-rebench's methodology, test set composition, or how it differs from Artificial Analysis, the magnitude of these shifts resists clean interpretation: they could reflect genuine capability differences in software engineering tasks, calibration differences between benchmarks, or selection effects in which models were evaluated on which benchmark.

Cole Brennan