The Inference Report

Claude Code holds the top position on SWE-rebench at 52.9%, with Junie at 52.1% and Claude Opus 4.6 at 51.7%, but the most striking pattern appears in the Artificial Analysis benchmark where the rankings have shifted substantially since the prior update: Claude Opus 4.5 dropped from position 8 (49.7) to position 12 (43.8), GLM-5 fell from position 7 (49.8) to position 15 (42.1), and Kimi K2.5 plummeted from position 13 (46.8) to position 19 (37.9), while Kimi K2 Thinking climbed from position 28 (40.9) to position 13 (43.8) and GLM-4.6 rose from position 54 (32.5) to position 22 (37.1). DeepSeek V3.2 Speciale dropped sharply from position 46 (34.1) to position 66 (29.4), the most dramatic reversal in the upper tier. The scale of these movements raises questions about benchmark stability: shifts of 5 to 10 percentage points across a single update are large enough to suggest either meaningful model degradation, evaluation methodology changes, or dataset variance rather than genuine performance evolution. Kimi K2 Thinking's 3-position rise paired with K2.5's 6-position fall is particularly difficult to interpret without visibility into what changed in the evaluation protocol. On SWE-rebench, the top tier remains essentially frozen, which either reflects genuine consolidation at the capability ceiling or indicates that benchmark has reached saturation where further discrimination is difficult. The Artificial Analysis benchmark, by contrast, shows volatility that demands scrutiny: if these are the same models tested under consistent conditions, such reversals warrant investigation into whether scoring criteria, test set composition, or model access states have shifted between runs.

Cole Brennan