The Inference Report

The SWE-rebench rankings remain static across the top tier, with AnthropicFable 5 holding 64.5% and the top ten models unchanged, suggesting either stable evaluation conditions or that recent model releases haven't yet displaced the established leaders in coding task resolution. By contrast, Artificial Analysis shows substantial churn: Muse Spark Open enters at #34 with 55.8, pushing prior models down one position each; Solar Pro 4 drops from #67 to #75 despite unchanged performance at 26.6; Nemotron 3.5 Super VL and Nemotron 3.5 Bolt 74B-A9B appear as new entries at #76 and #124 respectively, while Gemma 3 12B Instruct falls dramatically from #472 to #475 with a score drop from 3.8 to 3.8, a data anomaly suggesting either scoring methodology change or evaluation inconsistency at the lower end. The coding benchmark shows no movement, which raises a methodological question: either the variance intervals (ranging from ±0.54% to ±1.83%) are wide enough to absorb real performance shifts, or the benchmark has reached saturation at the top where further differentiation is difficult. The Artificial Analysis churn at positions 34 and beyond appears driven by new model releases rather than score improvements in existing models, indicating the two benchmarks measure different capability dimensions or operate on different evaluation cadences. Without historical score changes for existing models in SWE-rebench, it's unclear whether the stability reflects genuine plateau or insufficient temporal resolution to detect drift.

Cole Brennan