The Inference Report

Claude Code holds the top position on SWE-rebench at 52.9%, unchanged from the previous cycle, while a new entrant, Junie, enters at 52.1% in second place. Claude Opus 4.6 dropped from 51.7% to a tie at 51.7% with gpt-5.2-2025-12-11-xhigh, though its ranking shifted down one position as the GPT variant pulled ahead in the secondary ordering. The most notable movement occurs lower in the rankings: Claude Opus 4.5 fell sharply from 49.7% on Artificial Analysis (position 8) to 43.8% on SWE-rebench (position 12), a 6-point gap that suggests either a real decline in code-solving ability or a meaningful divergence in what these two benchmarks measure. GLM-5 similarly dropped from 49.8% to 42.1%, and Kimi K2.5 fell from 46.8% to 37.9%, indicating that the Artificial Analysis benchmark may weight certain model capabilities differently than SWE-rebench does. Conversely, Kimi K2 Thinking gained 3 positions, rising from 40.9% to 43.8%, and GLM-4.6 climbed from 32.5% to 37.1%, suggesting these models may have received updates or that SWE-rebench captures their strengths more clearly. The top tier remains clustered between 52.9% and 51.0%, with no model breaking into the low 53% range, indicating a plateau in performance on this benchmark's task distribution rather than rapid improvement. Without access to the evaluation methodology details for either benchmark, the divergence between them warrants scrutiny: SWE-rebench appears to penalize certain architectures or reasoning approaches that Artificial Analysis credits, though both benchmarks are evaluating the same underlying problem space of software engineering tasks.

Cole Brennan