The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, unchanged from the previous cycle, while the tier immediately below has compressed: gpt-5.2-2025-12-11-medium, GLM-5, and gpt-5.4-2026-03-05-medium now cluster between 62.8% and 64.4%, with GLM-5 climbing from #7 to #3 and gaining 13 percentage points since the Artificial Analysis benchmark. Gemini 3.1 Pro Preview dropped from #2 to #5 on SWE-rebench despite scoring 62.3%, a 5.1-point gain over its Artificial Analysis score of 57.2, suggesting the two benchmarks measure different problem distributions or evaluation rigor. Kimi K2.5 and Kimi K2 Thinking both posted substantial gains, 12.5 and 16.5 points respectively on Artificial Analysis, and moved up the SWE-rebench ranks to #13 and #17, though the magnitude of improvement raises questions about whether the models were retrained, fine-tuned on benchmark data, or if the evaluation protocols differ materially between the two systems. The broader pattern shows Claude models and GPT variants dominating the top ten on SWE-rebench while Chinese models (GLM-5, Kimi variants, Qwen lines) have narrowed the gap, and the divergence between SWE-rebench and Artificial Analysis rankings for several mid-tier models suggests these benchmarks are not interchangeable proxies for coding ability. MiMo-V2-Omni dropped from the Artificial Analysis rankings entirely despite previously scoring 43.4, a notable exit that warrants clarification on whether the model was discontinued or simply failed to meet evaluation criteria this cycle.

Cole Brennan