The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, unchanged from the previous cycle, while the tier immediately below has compressed considerably. GPT-5.2-2025-12-11-medium sits at 64.4%, with GLM-5, GPT-5.4-2026-03-05-medium, GLM-5.1, and Gemini 3.1 Pro Preview clustered between 62.3 and 62.8 percent, suggesting a plateau where further gains require either methodological refinement or fundamentally different approaches to coding tasks. The notable movers are Claude Sonnet 4.6, which jumped from position 11 to 8 with a 9-point gain from 51.7 to 60.7 percent, and GLM-4.7, which climbed from position 43 to 14 by improving 16.6 points from 42.1 to 58.7 percent. Gemini 3.1 Pro Preview declined from position 3 to 6, dropping 4.9 points from 57.2 to 62.3 percent on SWE-rebench despite maintaining strong performance on Artificial Analysis. The divergence between these two benchmarks is instructive: Claude Opus 4.6 scores 65.3 on SWE-rebench but only 53 on Artificial Analysis, a 12.3-point gap, while Gemini 3.1 Pro Preview shows narrower separation at 62.3 versus 57.2, suggesting the benchmarks weight different problem classes or that SWE-rebench's evaluation criteria reward the specific capabilities Claude's latest iteration emphasizes. Without visibility into whether SWE-rebench's test set, scoring rubric, or evaluation harness changed, it remains unclear whether these movements reflect genuine capability shifts or benchmark sensitivity to model-specific strengths.

Cole Brennan