The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, a 12.3-point jump from its previous ranking of 53% on Artificial Analysis, though the two benchmarks measure different problem sets and cannot be directly compared. The top six models on SWE-rebench cluster between 62.3% and 65.3%, with gpt-5.2-2025-12-11-medium at 64.4%, GLM-5 and gpt-5.4-2026-03-05-medium tied at 62.8%, GLM-5.1 at 62.7%, and Gemini 3.1 Pro Preview at 62.3%, suggesting convergence in code completion capability at the frontier. Gemini 3.1 Pro Preview dropped from second place on Artificial Analysis (57.2) to sixth on SWE-rebench (62.3), indicating the benchmarks reward different model properties, with SWE-rebench apparently favoring systems trained or optimized specifically for repository-level code repair. GLM-4.7 advanced from rank 36 on Artificial Analysis (42.1) to rank 14 on SWE-rebench (58.7), a 16.6-point gain, while Kimi K2.5 climbed from rank 21 (46.8) to rank 16 (58.5), and Kimi K2 Thinking jumped from rank 44 (40.9) to rank 21 (57.4), suggesting these models contain architectural or training choices that translate effectively to the SWE-rebench evaluation methodology. The Artificial Analysis leaderboard shows no movement in the top 30 positions relative to the previous snapshot, indicating ranking stability at the frontier when measured on that benchmark. SWE-rebench's methodology, evaluating models on real pull requests and repository contexts, appears more sensitive to model-specific optimizations than Artificial Analysis, which may employ broader capability assessment; without detailed documentation of task overlap or divergence, claims about which benchmark better predicts real-world code repair performance remain speculative.

Cole Brennan