The Inference Report

On SWE-rebench, the top tier remains stable with gpt-5.5-2026-04-23-xhigh holding first place at 62.7% and the next five positions unchanged, but meaningful movement appears below that line: Gemini 3.1 Pro Preview dropped from 57.2% to 51.1% on Artificial Analysis (down six positions), Gemini 3.5 Flash fell from 55.3% to 49.5% on SWE-rebench and 55.3% to 50.2% on Artificial Analysis, and Kimi K2.6 declined from 53.9% to 46.5% on Artificial Analysis while holding steady on SWE-rebench. GLM-4.7 improved notably from 42.1% to 50.7% on Artificial Analysis, moving into the top 20, and GLM-4.7 itself advanced from 42.1% to 50.7% on Artificial Analysis, though it remains at 38.2% on SWE-rebench. The Artificial Analysis leaderboard shows broader volatility: Claude Fable 5 dropped from 64.9 to 59.9, Claude Opus 4.8 fell from 61.4 to 55.7, and GPT-5.5 declined from 60.2 to 54.8, suggesting either a recalibration of the benchmark methodology or systematic changes in model evaluation conditions. Lower-ranked models show the largest percentage-point losses across both benchmarks, with many models in the 100-200 range losing 5-8 points, raising the question of whether this reflects actual model degradation, benchmark recalibration, or environmental factors like inference conditions that affect consistency. The SWE-rebench scores remain tighter and more stable than Artificial Analysis, which could indicate either greater robustness in that benchmark's methodology or a narrower evaluation scope that leaves less room for variance. Without clarity on whether these benchmarks measure identical task sets or use different evaluation protocols, the divergence between the two makes it difficult to assess whether the movement represents genuine capability shifts or measurement artifacts.

Cole Brennan