The Inference Report

The SWE-rebench leaderboard shows compression at the top but no meaningful movement within the tested range. Claude Code, Junie, and Claude Opus 4.6 remain locked in the 51.7 to 52.9 percent band, with the top six models separated by less than a percentage point, indicating a plateau in discriminative power rather than genuine progress. Below that tier, ranking shifts are notable but scores tell a different story: Claude Opus 4.5 dropped from 49.7 to 43.8 on Artificial Analysis while holding at 43.8 on SWE-rebench, GLM-5 fell from 49.8 to 42.1, and Kimi K2.5 collapsed from 46.8 to 37.9, all suggesting either benchmark recalibration or evaluation variance rather than model regression. Conversely, Kimi K2 Thinking jumped 2.9 points to 43.8 on SWE-rebench and GLM-4.6 gained 4.6 points to 37.1 on Artificial Analysis, but these gains occur in a region where single-digit improvements are routine and may reflect test set sensitivity rather than architectural breakthroughs. The two benchmarks diverge substantially in their top rankings (GPT-5.4 and Gemini 3.1 Pro Preview both score 57.2 on Artificial Analysis versus Claude Code's 52.9 on SWE-rebench), raising questions about what each evaluates: SWE-rebench appears stricter or tests different problem classes, making direct comparison unreliable. Without documentation of methodology changes, evaluation set stability, or statistical confidence intervals, the apparent volatility in mid-tier positions cannot be distinguished from noise. The real finding is not movement but stagnation at the frontier and inconsistency across benchmarks, both of which limit confidence in using either as a proxy for practical coding capability.

Cole Brennan