The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, a 12.3-point jump from its previous ranking of 53 on Artificial Analysis, though the two benchmarks measure different problem sets and cannot be directly compared. The top tier has consolidated around 62-65% on SWE-rebench, with GPT-5.2-2025-12-11-medium at 64.4% and three models tied at 62.8% (GLM-5, Junie, GPT-5.4-2026-03-05-medium), suggesting diminishing returns in coding task performance at the frontier. More striking are the mid-tier movements: GLM-5 climbed from position 16 to 3 on SWE-rebench, GLM-4.7 rose from 40 to 14, and Kimi K2.5 advanced from 26 to 16, indicating that Chinese model families have made substantial gains on this particular benchmark. Gemini 3.1 Pro Preview, by contrast, dropped from third on Artificial Analysis (57.2) to seventh on SWE-rebench (62.3), a relative decline despite the absolute score increase, which may reflect task-specific strengths rather than regression. On Artificial Analysis, the leaderboard remains fluid with 33 new entries across the 373-model roster, including several reasoning-focused variants and smaller parameter models, though the top ten remain dominated by GPT and Claude variants. The SWE-rebench benchmark appears more selective and stable, with only 34 models tracked versus hundreds on Artificial Analysis, making it a tighter measure of coding capability but limiting visibility into broader model performance distributions. Without methodological details on how SWE-rebench tasks differ from Artificial Analysis's evaluation protocol, the divergence in model rankings suggests these benchmarks may reward different architectural or training choices, a distinction worth investigating rather than treating the benchmarks as interchangeable measures of coding prowess.

Cole Brennan