The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, unchanged from the previous cycle, while the tier immediately below has solidified around 62 to 64 percent with gpt-5.2-2025-12-11-medium at 64.4% and GLM-5 and gpt-5.4-2026-03-05-medium both at 62.8%. The meaningful shifts appear in models that have climbed several positions: Claude Opus 4.6 rose from rank 4 to rank 1 (a gain of 12.3 points from 53 on the Artificial Analysis benchmark), GLM-5 advanced from rank 7 to rank 3 (13 points), Kimi K2.5 jumped from rank 16 to rank 13 (11.7 points), and Kimi K2 Thinking moved from rank 35 to rank 17 (16.5 points). Gemini 3.1 Pro Preview slipped from rank 2 to rank 5, losing 5.1 points despite remaining competitive. The SWE-rebench methodology appears to diverge notably from the Artificial Analysis scores, particularly for Claude and Kimi models, which suggests the benchmarks may weight different problem classes or evaluation criteria differently. The gap between first and tenth place on SWE-rebench is 5.7 percentage points, indicating a tightening at the top end, while the Artificial Analysis leaderboard shows a steeper spread, with the top model at 57.2 and rank 10 at 49.2. Without access to the specific evaluation methodology differences between these two benchmarks, it is unclear whether the SWE-rebench gains reflect genuine capability improvements or reflect a benchmark that rewards different architectural or training choices.

Cole Brennan