The Inference Report

The SWE-rebench rankings show no movement in the top tier: AnthropicFable 5 holds first at 64.5% ± 1.41%, GrokGrok 4.5 remains second at 63.8% ± 0.60%, and the next five positions remain unchanged through OpenAIGPT-5.6 Sol at 62.3% ± 1.83%. The Artificial Analysis benchmark, by contrast, underwent substantial reorganization across its full 467-model roster, with NVIDIA Nemotron 3 Nano 4B dropping from position 321 to 465, and multiple models in the 320s range shifting position by single or double digits. The SWE-rebench evaluation tests code generation agents on real GitHub issues with controlled execution environments, while Artificial Analysis appears to assess a broader range of model capabilities across a much larger set of entries, making direct comparison difficult. The stability of the SWE-rebench top tier suggests either that these models have reached a performance plateau on that particular task or that the evaluation's variance bands (ranging from ±0.54% to ±1.83%) are narrow enough to prevent ranking churn. The extensive reshuffling in Artificial Analysis, which uses point scores without reported confidence intervals, may reflect either more sensitive discrimination between models or differences in evaluation methodology. Without access to the specific problem distributions, success criteria, or statistical methodology behind each benchmark, it remains unclear whether the SWE-rebench stability reflects genuine performance plateaus or whether the benchmark's design constrains differentiation among top performers.

Cole Brennan