The Inference Report

The SWE-rebench rankings show modest churn at the top tier but reveal significant instability in the middle and lower ranks on Artificial Analysis. On SWE-rebench, the top five remain locked: gpt-5.5-2026-04-23-xhigh holds 62.7%, followed by Codex (60.4%), Claude Code (59.6%), gpt-5.5-2026-04-23-medium (58.9%), and gpt-5.4-2026-03-05-medium (54.9%), with no score changes from prior results. Below that tier, Claude Opus 4.7 and Gemini 3.1 Pro Preview each dropped roughly 4 points (to 53.1% and 51.1% respectively), while Kimi K2.6 fell from 53.9% to 46.5%, a 7.4-point decline that suggests either methodology drift or model degradation rather than genuine progress. Across the Artificial Analysis leaderboard, the data is identical to the prior snapshot, indicating no new evaluations or score recalculations occurred. The absence of movement in a 383-model ranked list, combined with the stability of the top SWE-rebench performers, suggests these benchmarks may be operating on different evaluation cadences or that the SWE-rebench methodology itself is in flux. The meaningful signal here is negative: large score drops like Kimi K2.6's warrant investigation into whether the benchmark conditions changed, whether the model was updated, or whether previous scores were inflated. Without evidence of fresh evaluation runs, neither leaderboard documents actual progress this cycle.

Cole Brennan