The Inference Report

The SWE-rebench leaderboard shows stability at the top with no score changes, while Artificial Analysis reveals significant churn in the middle tier. On SWE-rebench, the top six positions remain locked: GPT-5.5-xhigh holds 62.7 percent, Junie Agent at 61.6 percent, and Codex Agent at 60.4 percent, with confidence intervals tight enough that these gaps reflect real differences in code-fixing capability. The benchmark's methodology appears sound for the top tier, where agentic systems and specialized models cluster in the 53-62 percent range, but the tail end shows concerning compression, with Gemma 4 31B dropping to 16.5 percent from a larger gap that suggests either ceiling effects or evaluation brittleness on edge cases. On Artificial Analysis, the picture diverges sharply: Claude Fable 5 now leads at 59.9, displacing GPT-5.5 to third at 54.8, yet this benchmark lacks the technical transparency of SWE-rebench and the score ranges (59.9 to 1.0) suggest different calibration or problem distribution. GLM-4.6 jumped from 99th to 84th with a 3.6-point gain, while Ling 2.6 Flash dropped from 137th to 181st, losing 5.2 points, indicating either test set variance or model instability rather than methodological drift. The discrepancy between benchmarks matters: SWE-rebench isolates code-solving ability through controlled problem solving, while Artificial Analysis appears to measure broader capability, making direct ranking comparison misleading. Neither benchmark documents how they handle partial solutions, timeout handling, or whether they weight problem difficulty, which limits confidence in interpreting movements below the top ten.

Cole Brennan