The Inference Report

Claude Opus 4.6 holds the top position on SWE-rebench at 65.3%, a gain of 12.3 points from its previous score of 53 on Artificial Analysis, while the Artificial Analysis leaderboard shows minimal movement in the top tier, with GPT-5.4 and Gemini 3.1 Pro Preview both at 57.2. The divergence between the two benchmarks reflects a known challenge in LLM evaluation: SWE-rebench and Artificial Analysis measure different problem distributions and solution strategies, making direct comparisons across methodologies unreliable. On SWE-rebench, the clustering is tight in the top ten, with only 5.7 percentage points separating first from tenth, suggesting the benchmark may be reaching saturation for frontier models or that the test set lacks discriminative power at the high end. GLM-5 and Kimi K2.5 show substantial gains on SWE-rebench (13 and 9.7 points respectively), yet their Artificial Analysis positions remain largely stable, indicating these models may have specialized improvements for code-related tasks rather than across-the-board capability increases. The Artificial Analysis benchmark, which covers a broader evaluation surface, shows Gemma 4 31B and Gemma 4 E4B entering the top 150, suggesting incremental progress in the open-source tier, though the majority of entries below rank 40 shuffle positions without meaningful score changes. Neither benchmark provides sufficient methodological transparency in the data to assess whether score improvements reflect genuine capability gains or dataset-specific optimization, and the absence of error bars or confidence intervals prevents determination of statistical significance for most movements.

Cole Brennan