The Inference Report

The SWE-rebench top tier remains unchanged from the previous cycle, with Claude Code holding 52.9%, Junie at 52.1%, and Claude Opus 4.6 and gpt-5.2-2025-12-11-xhigh tied at 51.7%, suggesting a plateau in incremental gains among the highest-performing models. Below this ceiling, however, significant volatility emerges: Gemini 3 Pro Preview dropped from 48.4 to 46.7 on Artificial Analysis while maintaining rank #8 on SWE-rebench, whereas Kimi K2 Thinking climbed from 40.9 to 43.8 on Artificial Analysis and rose from #33 to #13 on SWE-rebench, a 20-position jump that points to either methodological divergence between the two benchmarks or genuine capability shifts in specific coding tasks. GLM-5 fell sharply from 49.8 to 42.1 on Artificial Analysis while dropping from #7 to #15 on SWE-rebench, a discrepancy that warrants scrutiny into whether the Artificial Analysis evaluation captures different problem distributions or if the SWE-rebench methodology has tightened. Kimi K2.5 presents the inverse pattern, declining from 46.8 to 37.9 on SWE-rebench but remaining at 46.8 on Artificial Analysis, suggesting the two benchmarks reward different architectural or prompt-handling strategies. The broader pattern indicates that neither benchmark is settling into stable rankings: models in the 35-50% range on SWE-rebench show rank swings of 10-20 positions across cycles, and the divergence between SWE-rebench and Artificial Analysis scores (sometimes 5-10 percentage points) suggests these are measuring meaningfully different aspects of code generation capability rather than converging on a unified signal.

Cole Brennan