The Inference Report

The SWE-rebench leaderboard shows no movement from the previous snapshot, with OpenAI's gpt-5.5-2026-04-23-xhighModel holding 62.7% (±0.91%) at the top, followed by JunieJunieAgent at 61.6% (±0.64%) and OpenAICodexAgent at 60.4% (±1.37%). The Artificial Analysis benchmark, however, exhibits substantial churn across its 409 entries, with Motif 3 entering at rank 22 (44.1 points) and Mercury 2 dropping from rank 109 to 128 (25.3 to 21.4 points), suggesting either model updates or changes in evaluation methodology that warrant clarification. The lack of overlap between top performers on the two benchmarks, Claude Fable 5 leads Artificial Analysis at 59.9 while ranking nowhere on SWE-rebench's top 24, indicates these measure distinct problem spaces rather than equivalent capabilities. SWE-rebench's narrow confidence intervals (0.45% to 1.98%) suggest tighter experimental control than typical leaderboards, though the static rankings across both snapshots raise questions about evaluation frequency and whether these represent live benchmarks or periodic releases. Without details on SWE-rebench's test set composition, task distribution, or whether agents can call external tools, it remains unclear whether the 1.1-point gap between first and second place reflects genuine capability differences or measurement noise within the confidence bounds.

Cole Brennan