The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, unchanged from the previous cycle, while the second tier has tightened considerably around 62-64 percent with gpt-5.2-2025-12-11-medium, GLM-5, gpt-5.4-2026-03-05-medium, and GLM-5.1 all clustering within a single point. The most significant movement appears in the mid-tier: GLM-4.7 climbed from rank 42 at 42.1 percent to rank 14 at 58.7 percent, a gain of 16.6 points; Kimi K2.5 jumped from rank 27 at 46.8 percent to rank 16 at 58.5 percent; and Kimi K2 Thinking advanced from rank 51 at 40.9 percent to rank 21 at 57.4 percent. These represent genuine improvements in coding task resolution, not ranking artifacts. Gemini 3.1 Pro Preview dropped from rank 3 to rank 6 despite holding 62.3 percent, reflecting the compression of scores in the upper tier rather than performance degradation. On the Artificial Analysis benchmark, the rankings remain relatively stable at the extreme ends, though GPT-5.5 continues to lead at 60.2 and Claude Opus 4.7 sits at 57.3. The divergence between SWE-rebench and Artificial Analysis scores is pronounced: Claude Opus 4.6 scores 65.3 percent on SWE-rebench but only 53 percent on Artificial Analysis, suggesting the benchmarks measure distinct capabilities or that SWE-rebench may have different task difficulty distribution. Without visibility into whether the SWE-rebench test set changed or models were simply retested, the GLM and Kimi improvements warrant scrutiny regarding whether they reflect algorithmic advances or evaluation variance.

Cole Brennan