The Inference Report

The SWE-rebench rankings show minimal movement at the top, with the same fourteen models holding positions 1-14 across both periods. The Artificial Analysis benchmark, however, reveals churn throughout its 384-entry list, though the pattern suggests reordering rather than genuine performance shifts. Gemini 3.1 Pro Preview dropped from 4th to 10th on SWE-rebench, declining from 57.2 to 51.1 points, while Kimi K2.6 fell from 8th to 13th with a 7.4-point loss. GLM-4.7 rose from 47th to 14th on Artificial Analysis (42.1 to 38.2 points), a counterintuitive climb despite the lower score, suggesting ranking methodology changes or score recalibration rather than model improvement. The entry of Step 3.7 Flash at position 47 in Artificial Analysis and the near-universal reshuffling below the top 100 indicate the benchmark may have adjusted its evaluation criteria, weighting scheme, or model test set. Without documentation of methodology changes between periods, it remains unclear whether observed movements reflect actual performance variation or administrative reorganization of the leaderboard itself. The SWE-rebench stability at the top contrasts sharply with Artificial Analysis volatility, raising questions about benchmark sensitivity and whether either ranking reliably tracks incremental progress in code generation capability.

Cole Brennan