The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3 percent, unchanged from the previous cycle, while the tier below it has compressed significantly: gpt-5.2-2025-12-11-medium scores 64.4 percent, and a cluster of three models (GLM-5, Junie, and gpt-5.4-2026-03-05-medium) all sit at 62.8 percent. The most striking movement comes from GLM-5, which jumped from rank 17 at 49.8 percent on Artificial Analysis to rank 3 at 62.8 percent on SWE-rebench, a 13-point gain that suggests either a genuine capability improvement or a methodological difference between the two benchmarks worth examining. GLM-5.1 shows a similar trajectory, moving from rank 14 at 51.4 percent to rank 6 at 62.7 percent, and Kimi K2 Thinking climbed from rank 54 at 40.9 percent to rank 21 at 57.4 percent. Gemini 3.1 Pro Preview, by contrast, declined from rank 3 at 57.2 percent on Artificial Analysis to rank 7 at 62.3 percent on SWE-rebench, a modest slip that places it below several newer contenders despite holding strong absolute performance. The Artificial Analysis leaderboard shows GPT-5.5 leading at 60.2 percent with Claude Opus 4.7 at 57.3 percent and Gemini 3.1 Pro Preview at 57.2 percent, a different ordering entirely from SWE-rebench. This divergence between benchmarks raises a methodological question: SWE-rebench appears to reward certain architectural or training choices that Artificial Analysis does not, and without clarity on what each benchmark isolates, ranking movements alone cannot confirm whether progress is real or an artifact of evaluation design.

Cole Brennan