The Inference Report

On the SWE-rebench, the top tier remains stable with gpt-5.5-2026-04-23-xhigh holding 62.7%, Junie at 61.6%, and Codex at 60.4%, but movement in the middle ranks reveals both consolidation and fragmentation. Claude Sonnet 4.6 climbed from #8 to #10 while gaining 4.1 percentage points (47.2 to 51.3), and Gemini 3.1 Pro Preview moved from #9 to #11 with a 4.6-point increase (46.5 to 51.1), suggesting these models benefited from either test set changes or evaluation methodology shifts rather than architectural improvements alone. GLM-5.1's jump from #23 to #12 represents the most dramatic repositioning, rising 10.5 points from 40.2 to 50.7, which warrants scrutiny: either the model underwent substantial retraining or the benchmark's coding task distribution shifted to favor its strengths. Conversely, Gemini 3.5 Flash dropped from #7 to #13 despite a marginal score decline (50.2 to 49.5), a minor inversion that may reflect tighter clustering at this performance band. GLM-4.7 showed the largest absolute gain in the lower ranks, jumping from 33.8 to 38.2 across the two evaluations, though it remains at #17 on SWE-rebench. The Artificial Analysis benchmark, with its broader model coverage, presents a different ranking topology: Claude Fable 5 leads at 59.9, above GPT-5.5 at 54.8, inverting the SWE-rebench order and suggesting the two benchmarks weight different coding competencies or test different problem classes. Without disclosure of the evaluation methodology, task composition, test set overlap, execution environment, or whether SWE-rebench underwent revision, attributing these shifts to genuine capability differences versus benchmark drift remains uncertain. The consistency of top-tier models across both benchmarks provides some confidence in their relative ordering, but the volatility in middle ranks indicates either genuine model differentiation in narrow domains or measurement sensitivity that limits strong inference.

Cole Brennan