The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, a jump from sixth place and a gain of 12.3 percentage points from its previous 53 on Artificial Analysis, though the two benchmarks measure different problem sets and methodologies so direct comparison requires caution. The tier below it remains tightly clustered: gpt-5.2-2025-12-11-medium scores 64.4%, GLM-5 and gpt-5.4-2026-03-05-medium both reach 62.8%, and GLM-5.1 sits at 62.7%, with Gemini 3.1 Pro Preview at 62.3%. What distinguishes these movements is the compression at the top, five models now occupy a 2.6-point band, and the significant repositioning of Chinese models: GLM-5 advanced from rank 13 to rank 3 (49.8 to 62.8), GLM-4.7 jumped from 38 to 14 (42.1 to 58.7), and Kimi K2.5 rose from 23 to 16 (46.8 to 58.5). Gemini 3.1 Pro Preview's descent from second to sixth, despite a 5.1-point absolute gain to 62.3%, underscores how the benchmark shifted the entire distribution upward rather than revealing a single model's failure. The SWE-rebench methodology appears to reward architectural or training choices that these frontier models now share more evenly, particularly for code completion and repository-level problem solving. Whether this convergence reflects genuine capability parity or benchmark saturation, where the test's difficulty ceiling has been approached by multiple labs, remains an open question that requires examining the test construction and error analysis across models, not just the leaderboard positions.

Cole Brennan