The Inference Report

The top tier on SWE-rebench remains frozen: Claude Code holds 52.9%, Junie 52.1%, and the next four models cluster between 51.7% and 51.0%, with no movement in the top five positions. Below that line, the volatility increases sharply. Claude Opus 4.5 dropped from rank 8 to rank 12 while losing 5.9 percentage points (49.7 to 43.8), a decline that suggests either benchmark drift or a methodological shift in how the evaluation weights different problem categories. Kimi K2 Thinking gained 2.9 points and jumped from rank 34 to rank 13, the largest upward move in the dataset. Gemini 3 Pro Preview fell from rank 13 to rank 8 despite losing 1.7 points, a ranking shift driven by larger losses elsewhere. GLM-5 dropped 7.7 points (49.8 to 42.1) and fell from rank 15 to rank 7 in the Artificial Analysis benchmark, while on SWE-rebench it moved from rank 15 to rank 15 with a smaller decline, suggesting the two benchmarks may measure different problem distributions or that GLM-5's performance on specific task categories degraded significantly. Kimi K2.5 fell 8.9 points on Artificial Analysis (46.8 to 37.9) and dropped from rank 19 to rank 19 on SWE-rebench, a rare case of consistent decline across both metrics. The data shows no clear pattern of across-the-board improvement; instead, individual models show both gains and losses, which raises questions about whether these benchmarks are tracking genuine capability shifts or whether evaluation procedures, test set composition, or model versioning have changed between measurement cycles.

Cole Brennan