The Inference Report

The SWE-rebench rankings remain static at the top, with OpenAI's gpt-5.5-2026-04-23-xhigh maintaining 62.7% and Junie's agent holding 61.6%, but the Artificial Analysis benchmark shows substantial movement in the mid-tier and below. DeepSeek R1 jumped from position 190 at 12.6% to position 144 at 18.5%, a gain of 5.9 percentage points that reflects either model improvement or evaluation methodology changes; Mistral Small 3.1 climbed from 255 at 8.6% to 172 at 14.7%, gaining 6.1 points across a 83-position swing. Claude 4 Sonnet dropped from 72 at 30.7% to 83 at 28.9%, losing 1.8 points despite holding its rank position number. Devstral Small 2 moved from 186 at 13.1% to 153 at 17.4%, and Llama 3.1 Instruct 8B climbed from 298 at 6.1% to 274 at 7.6%, both showing gains that suggest either these models were re-evaluated with different configurations or the benchmark itself shifted its evaluation criteria. The Artificial Analysis data spans 397 entries compared to 16 on SWE-rebench, creating an asymmetry in what constitutes meaningful movement: a 1-point swing at the top 10 of Artificial Analysis represents roughly 2 percent of the leader's score, while the same absolute change at position 350 is nearly a 20 percent relative improvement. New entry DiffusionGemma 26B A4B at position 185 with 13.5% provides no prior reference, making it impossible to assess whether this is a newly evaluated model or a previously omitted one. Without access to methodology details for either benchmark, the interpretation of these shifts remains constrained to surface observation: SWE-rebench appears stable and possibly closed to new entries, while Artificial Analysis exhibits churn consistent with either rolling re-evaluation or score recalibration across the full roster.

Cole Brennan