The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, unchanged from the previous round, while the Artificial Analysis benchmark shows Claude Opus 4.7 at 57.3 in first place, suggesting the two benchmarks are measuring different problem distributions or difficulty levels. The SWE-rebench leaderboard has consolidated around a narrow band: the top six models cluster between 65.3% and 62.3%, with gpt-5.2-2025-12-11-medium at 64.4% and GLM-5 and gpt-5.4-2026-03-05-medium both at 62.8%, indicating diminishing returns as models approach saturation on this evaluation set. Notable climbers on Artificial Analysis include Kimi K2.6 entering at position 4 with 53.9 points and JT-MINI appearing at position 113 with 25.4 points, though their SWE-rebench performance is not reported, making cross-benchmark validation difficult. Gemini 3.1 Pro Preview dropped from second place on Artificial Analysis (57.2) to sixth on SWE-rebench (62.3), a reversal that warrants scrutiny of the underlying tasks, SWE-rebench may emphasize code generation or repository manipulation where Claude and GPT variants perform better, while Artificial Analysis may weight reasoning or planning more heavily. The SWE-rebench methodology itself remains opaque in the provided data; without visibility into task design, evaluation protocol, or whether scores are statistically independent, it is unclear whether the tight clustering reflects genuine convergence in model capability or whether the benchmark has begun to saturate as a discriminator. The two-benchmark divergence suggests practitioners should verify performance on their specific use case rather than treating either leaderboard as a universal proxy for software engineering capability.

Cole Brennan