The Inference Report

On SWE-rebench, the top tier remains stable: OpenAI's gpt-5.5-2026-04-23-xhighModel holds 62.7% (±0.91%), followed by JunieJunieAgent at 61.6% (±0.64%) and OpenAICodexAgent at 60.4% (±1.37%), with confidence intervals that do not overlap meaningfully enough to suggest ranking instability. The spread between positions 1 and 24 spans from 62.7% to 16.5%, a 46-point gap that reflects genuine capability stratification rather than noise. What merits scrutiny is the benchmark's design: SWE-rebench evaluates agent-based code completion on real software engineering tasks, a more applied setting than many token-prediction benchmarks, yet the evaluation methodology, particularly how task success is defined and whether partial credit is awarded, remains underspecified in the data provided. On Artificial Analysis, the roster has undergone substantial motion: Gemini 3.6 Flash enters at rank 15 (50.1 points), displacing prior entries downward, while Trinity Large Thinking drops from rank 112 (24.5 points) to rank 157 (18.2 points), a 6.3-point decline suggesting either a recalibration of the benchmark or a shift in evaluation conditions. The Artificial Analysis leaderboard exhibits denser clustering in the 30-50 point range than SWE-rebench, with many models separated by tenths of a point; this density raises questions about whether differences of 0.3-0.5 points reflect reproducible capability gaps or measurement variance. Neither benchmark's methodology clarifies whether scores are averaged across multiple runs, how task selection bias is controlled, or whether confidence intervals reflect statistical significance or merely reported uncertainty bounds. The two benchmarks show weak rank correlation at the extremes, Claude Fable 5 ranks first on Artificial Analysis (59.9) but does not appear on the SWE-rebench top 24, while gpt-5.5-xhigh dominates SWE-rebench but ranks sixth on Artificial Analysis (54.8), suggesting they measure distinct problem classes or that one benchmark's evaluation is more sensitive to architectural choices that do not generalize.

Cole Brennan