The Inference Report

Claude Opus 4.6 jumped from fourth to first on SWE-rebench, posting 65.3% versus its previous 53%, while gpt-5.2-2025-12-11-medium climbed to second at 64.4%, up from 51.0%. GLM-5 reached third at 62.8% after sitting at 49.8%, and DeepSeek-V3.2 vaulted from twenty-first at 37.5% to sixth at 60.9%, a 23.4-point gain that represents the largest movement in the cohort. The top tier has consolidated upward across the board: Gemini 3.1 Pro Preview slipped to fifth despite improving from 57.2% to 62.3%, a reversal that signals the benchmark itself has tightened or recalibrated rather than merely reflecting model improvement. Claude Code, which led the prior ranking at 52.9%, now sits fourteenth at 58.4%, and Junie fell from second at 52.1% to eleventh at 59.5%, both absolute gains masked by steeper climbs elsewhere. The Artificial Analysis benchmark shows less volatility: GPT-5.4 and Gemini 3.1 Pro Preview tie at 57.2%, unchanged from the previous period, while Claude Opus 4.6 holds at 53 in that metric despite its 12.3-point leap on SWE-rebench. The divergence between benchmarks suggests SWE-rebench scores are shifting faster than Artificial Analysis scores, which may reflect either different evaluation methodologies, different task distributions, or recalibration of the SWE-rebench itself. Twelve new entries appeared in the SWE-rebench top twenty-eight, including gpt-5.4-2026-03-05-medium at fourth and Qwen3.5-397B-A17B at ninth, while multiple prior top performers dropped entirely, indicating substantial churn in which models are being evaluated. Without access to the specific SWE-rebench problem set or scoring methodology changes between periods, the magnitude of these shifts warrants scrutiny: improvements of 10 to 20 points across the board could reflect genuine model advances, or they could signal that the benchmark has been modified, expanded, or re-baselined in ways that affect comparability.

Cole Brennan