The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, a gain of 12.3 percentage points from its previous ranking of 53% on Artificial Analysis, though these are separate benchmarks measuring different aspects of model capability. On SWE-rebench specifically, the top tier has compressed into a narrow band: gpt-5.2-2025-12-11-medium at 64.4%, GLM-5 and gpt-5.4-2026-03-05-medium tied at 62.8%, and GLM-5.1 at 62.7% occupy positions two through five, with no model scoring below 60% until rank seven. The more significant movement appears in the middle ranks, where GLM-4.7 jumped from position 38 (42.1 on Artificial Analysis) to position 14 (58.7% on SWE-rebench), Kimi K2.5 advanced from position 23 (46.8) to position 16 (58.5%), and Kimi K2 Thinking climbed from position 46 (40.9) to position 21 (57.4%), suggesting these models either improved substantially or benefit from SWE-rebench's evaluation methodology relative to Artificial Analysis scoring. Gemini 3.1 Pro Preview dropped from position 2 on Artificial Analysis to position 6 on SWE-rebench, declining from 57.2 to 62.3%, a counterintuitive result that may reflect differences in how the benchmarks weight problem-solving approaches or test coverage. The SWE-rebench results show less volatility at the extremes than Artificial Analysis, with the bottom ranks similarly stable, but the compression at the top and selective jumps in the middle suggest SWE-rebench either captures a narrower slice of coding ability or rewards specific architectural choices that certain model families exploit more effectively.

Cole Brennan