The Inference Report

The SWE-rebench rankings show Claude models displacing earlier leaders through variant proliferation rather than uniform improvement. Claude Opus 4.8-xhigh entered at 56.4% (rank 5), Claude Opus 4.7-high at 53.1% (rank 7), and Claude Sonnet 4.6-high at 51.3% (rank 9), all marked as new entries, which suggests these represent configuration variants of existing models rather than new releases. The top tier remains stable: gpt-5.5-2026-04-23-xhigh holds 62.7%, Codex 60.4%, and Claude Code 59.6%. Below the leaders, Gemini 3.1 Pro Preview dropped from 57.2 on Artificial Analysis to 51.1 on SWE-rebench (rank 10), a 6.1-point gap that flags a discrepancy between the two benchmarks worth investigating. Kimi K2.6 fell from 53.9 to 46.5 (rank 13), and GLM-4.7 declined from 42.1 to 38.2 (rank 14), suggesting these models either perform materially worse on coding tasks specifically or that SWE-rebench's evaluation criteria diverge meaningfully from Artificial Analysis's methodology. The Artificial Analysis leaderboard itself shows no movement in the top tier and remains dominated by Claude Opus 4.8 (61.4) and GPT-5.5 (60.2), with the field compressed tightly between ranks 1 and 20. Without access to SWE-rebench's exact task distribution, evaluation protocol, or whether it measures pass rates, time-to-solution, or other criteria, the divergence between the two benchmarks cannot be fully explained, but the pattern suggests they are testing distinct problem classes or applying different scoring thresholds.

Cole Brennan