The Inference Report

Claude Opus 4.6 holds the SWE-rebench lead at 65.3%, a 12.3-point gain from its prior position of 53 on Artificial Analysis, while Gemini 3.1 Pro Preview dropped from first place on Artificial Analysis (57.2) to fifth on SWE-rebench (62.3), suggesting the two benchmarks measure different aspects of code generation capability or apply distinct evaluation criteria. GLM-5 and Kimi K2.5 both climbed sharply on SWE-rebench relative to their Artificial Analysis rankings, with GLM-5 jumping from 49.8 to 62.8 and Kimi K2.5 from 46.8 to 58.5, indicating either improved performance on repository-level tasks or a methodology that rewards certain architectural choices more than point-estimate evaluations do. The top five models on SWE-rebench (Claude Opus 4.6, gpt-5.2-2025-12-11-medium, GLM-5, gpt-5.4-2026-03-05-medium, and Gemini 3.1 Pro Preview) all score between 62.3 and 65.3 percent, a compressed band that contrasts with wider spreads in earlier rankings, though without visibility into SWE-rebench's task composition, evaluation protocol, or error bars, it remains unclear whether these clustering differences reflect genuine convergence in code-solving ability or artifacts of test set size and difficulty distribution. The Artificial Analysis list shows no new entries at position 310 (Llama 2 Chat 13B at 8.4), a minor data inconsistency that does not affect the overall ranking coherence. Both benchmarks show deep-model dominance, but the divergence between them in how frontier models rank underscores that code generation performance is not monolithic: repository-level problem-solving, as measured by SWE-rebench, appears to reward different capabilities than the aggregated tasks tracked by Artificial Analysis.

Cole Brennan