The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, unchanged from the previous evaluation, while the Artificial Analysis benchmark shows a more fragmented picture with GPT-5.5 leading at 60.2. On SWE-rebench, the top tier remains stable: gpt-5.2-2025-12-11-medium sits at 64.4%, GLM-5 and gpt-5.4-2026-03-05-medium are tied at 62.8%, and GLM-5.1 follows at 62.7%. The notable movements occurred in the mid-range, where GLM-4.7 climbed from rank 42 (42.1 points on Artificial Analysis) to rank 14 on SWE-rebench with 58.7%, GLM-5 jumped from rank 16 to rank 3, Kimi K2.5 advanced from rank 27 to rank 16, and Kimi K2 Thinking moved from rank 51 to rank 21. Gemini 3.1 Pro Preview dropped from rank 3 to rank 6 on SWE-rebench despite scoring 62.3%, suggesting the benchmark may be measuring distinct problem-solving dimensions than Artificial Analysis, where Gemini 3.1 Pro Preview ranks third at 57.2. The divergence between the two benchmarks is substantial: Claude Opus 4.6 scores 65.3% on SWE-rebench but only 53 on Artificial Analysis, a gap of 12.3 points. Across both benchmarks, no model appears to have regressed in absolute performance, though relative rankings shifted due to other models improving or being newly added to the evaluation. The SWE-rebench methodology, which focuses on software engineering tasks, appears to reward models differently than the broader Artificial Analysis evaluation, and without access to the specific test case changes or evaluation protocol updates, it remains unclear whether these movements reflect genuine capability shifts or methodological adjustments to the benchmark itself.

Cole Brennan