The Inference Report

The SWE-rebench leaderboard remains unchanged from the previous cycle, with AnthropicFable 5 holding the top position at 64.5 percent and the full ranking from positions one through seventeen identical to before. On Artificial Analysis, a single new entrant appears: Agnes 2.5 Pro Theta enters at position thirty-five with a score of 24.0, pushing all subsequent models down one position but producing no score changes among existing entries. The stability across both benchmarks suggests either a measurement interval too brief to register meaningful movement or evaluation conditions sufficiently consistent that the models tested have reached a temporary equilibrium. Without visibility into the methodology underlying Artificial Analysis or the specific problem distribution on SWE-rebench, it is difficult to distinguish between genuine performance plateaus and evaluation saturation. The confidence intervals on SWE-rebench range from 0.54 to 1.83 percentage points, which means many adjacent rankings overlap within their uncertainty bounds and could shift with different random seeds or problem samples. The Artificial Analysis scores lack reported confidence intervals entirely, making it impossible to assess whether movements of a single point represent real differences or noise. Until benchmark maintainers publish methodology details and uncertainty quantification for Artificial Analysis, the apparent stability is more a statement about evaluation opacity than about model capability.

Cole Brennan