The Inference Report

The SWE-rebench rankings show no movement at the top tier, with AnthropicFable 5, GrokGrok 4.5, and AnthropicOpus 5 holding positions 1 through 3 at 64.5%, 63.8%, and 63.4% respectively, their confidence intervals overlapping enough that the ordering reflects measurement noise rather than clear separation. Artificial Analysis data reveals two new entries in the top tier: Muse Wonder Full enters at position 3 with 55.8, displacing Claude Fable 5.1 down to position 4, and Muse Wonder debuts at position 7 with 52.4, both pushing other models down one slot. The Artificial Analysis benchmark differs structurally from SWE-rebench in both scale and composition, making direct correlation difficult; SWE-rebench focuses on software engineering tasks with uncertainty bands (standard deviations ranging from 0.54% to 1.83%), while Artificial Analysis reports single scores across a broader model landscape without stated methodology or error margins. The absence of methodological transparency in Artificial Analysis limits interpretation of whether Muse's rise reflects genuine capability gains or evaluation differences. Within SWE-rebench's narrower scope, the lack of score movement suggests either stable model performance or insufficient sample size to detect meaningful shifts, and the confidence intervals around top performers remain wide enough that ranking stability does not necessarily indicate stagnation. The two benchmarks appear to measure different problem spaces: SWE-rebench's high-performing models (64.5% to 63.4%) operate well above Artificial Analysis's top performer (57.6%), suggesting either task difficulty divergence or evaluation rigor differences that deserve scrutiny before drawing conclusions about the field's progress.

Cole Brennan