The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, up from a prior ranking of 9th at 53%, a gain of 12.3 percentage points that represents the largest single-model improvement visible in the current data. The top seven positions cluster tightly between 62.3% and 65.3%, with gpt-5.2-2025-12-11-medium at 64.4%, GLM-5 and Junie tied at 62.8%, gpt-5.4-2026-03-05-medium at 62.8%, GLM-5.1 at 62.7%, and Gemini 3.1 Pro Preview at 62.3%. On the Artificial Analysis benchmark, however, the rankings diverge sharply: GPT-5.5 leads at 60.2, while Claude Opus 4.6 ranks 9th at 53, suggesting the two benchmarks measure different capabilities or that SWE-rebench may weight certain problem classes differently. GLM-5 moved from 17th to 3rd on SWE-rebench (49.8 to 62.8), GLM-4.7 climbed from 44th to 14th (42.1 to 58.7), and Kimi K2 Thinking jumped from 54th to 21st (40.9 to 57.4) on the coding benchmark, indicating broad-based gains across Chinese models. Gemini 3.1 Pro Preview, by contrast, dropped from 3rd on Artificial Analysis (57.2) to 7th on SWE-rebench (62.3), a relative decline despite an absolute score increase, which may reflect that the coding-specific benchmark rewards different optimization choices than general-purpose evaluation. The lack of methodological detail in either benchmark limits confidence in interpreting these divergences: neither source discloses test set size, problem distribution, whether solutions are evaluated for correctness alone or for code quality, or how edge cases are handled, making it unclear whether the gap between Claude's dominance on SWE-rebench and its mid-tier position on Artificial Analysis reflects genuine capability differences or artifacts of evaluation design.

Cole Brennan