The Inference Report

May 21, 2026

The AI industry has abandoned the pretense that models are the product. Across infrastructure, benchmarks, and deployed systems, the consolidation pattern is unmistakable: the money, the compute, and the power are flowing toward agents, inference infrastructure, and the systems that make autonomous execution possible at scale.

Google has reorganized its entire product surface around agency. Gemini 3.5 Flash is engineered for agentic workflows, positioned as four times faster than Claude Opus 4.7 and twice as fast as Gemini 3.1 Pro. The company is unifying its coding tools under Antigravity and embedding agents across Search, Android, and enterprise platforms. Nvidia's CEO Jensen Huang announced a $200 billion market opportunity in CPUs for AI agents, not inference chips for chat. This reorientation is structural, not cosmetic. Google processes 3.2 quadrillion tokens per month. That metric is now the unit of metering, pricing, and control. Whoever owns the inference layer owns the billing relationship.

The compute arms race has become visible and expensive. xAI burned $6.4 billion in 2025 and is purchasing $2.8 billion in natural gas turbines over three years while paying Anthropic $1.25 billion per month for compute. Anthropic is on track for its first profitable quarter with $10.9 billion in projected Q2 revenue, a milestone neither OpenAI nor xAI has reached. OpenAI is preparing its IPO filing for as soon as September with Goldman Sachs and Morgan Stanley. These are capital-intensive utilities being valued as such. Figure AI's continuous livestream of humanoid robots handling packages is not marketing; it is proof of concept that the market will watch robots work. The question has shifted from whether agents will exist to who controls the compute they run on.

In benchmarks and deployed code, the shift manifests as concrete technical priorities. Claude Opus 4.6 climbed 12.4 points on SWE-rebench to reach 65.3 percent, the largest single-model improvement in the dataset, while GLM-5 and Kimi K2 Thinking each gained roughly 13 to 16 points. On GitHub, the trending patterns split between agentic coding frameworks that reduce hallucination and token waste, and unglamorous infrastructure: llama.cpp and whisper.cpp remain gravitational centers for efficient local inference, now joined by quantization and pruning strategies. The secondary wave addresses production realities: observability tools like Phoenix, time-series anomaly detection, data synthesis, and ML pipeline orchestration. These don't trend virally because they solve problems that only matter once something actually ships. Agentic coding promises leverage over writing itself. The infrastructure work promises leverage over everything that comes after.

Grant Calloway

AI LabsAll labs
From the WireAll feeds
Research Papers — FocusedAll papers
Simultaneous Envy and Equitability Guarantees cs.GT

Recent work in fair division has focused on either simultaneously satisfying closely related fairness notions or achieving a single notion across the ex-ante and ex-post worlds. We study the compatibility of two fundamentally different fairness notions: envy-freeness and equitability. For indivisible goods-only and chores-only settings, we study the existence and complexity of simultaneously satisfying their relaxations, revealing sharp contrasts between the two settings. We show that EF1+EQ1 may fail to exist even for normalized, additive valuations. Our main algorithmic result computes an EF1+EQ1 allocation for normalized binary goods with at most seven agents. In sharp contrast, binary chores admit the stronger EFX+EQX guarantee for any number of agents, even without normalization. We further initiate the study of cross-notion ex-ante--ex-post guarantees, asking whether randomized allocations can provide ex-ante guarantees for one notion while preserving ex-post guarantees for another.

Token-Level Advertising cs.GT

Generative AI is transforming how people access information, challenging traditional advertising mechanisms built around predefined slots. Towards generation-native advertising, we propose the Latent Advertiser Mixture Auction (LAMA), a token-level advertising mechanism that embeds advertiser influence directly into the generation process. Advertisers report local continuation values that induce advertiser-specific next-token policies, from which the platform decodes through a latent mixture while updating an allocation posterior. We show that LAMA satisfies Markov DSIC and IR, and achieves near-optimal KL-regularized welfare. We further develop a learning-based implementation that reconstructs the required reports online from learned local advantages and root values. Proof-of-concept experiments on real-world commercial-search query splits show that LAMA improves platform welfare and revenue while maintaining user-facing response quality, providing initial evidence for the feasibility of generation-native advertising.

Multilevel Fair Allocation under Additive Preferences cs.GT

We study multilevel fair resource allocation with tree-structured hierarchical relations among agents. At each level, the problem can be viewed locally as allocating an agent's bundle to its children, the overall allocation being a trace of this process iterated down to the leaves. Assuming that internal nodes' utilities are the utilitarian welfare of their children, and the leaves have classical additive utilities over items, we first propose multilevel adaptations of usual envy-based fairness notions (e.g., WEF1). We present three adaptations and show that the choice among them is not neutral. We prove that, under identical preferences, the three adapted envy-based notions coincide, and that the Multilevel extension of Weighted Round Robin (Chakraborty et al., 2021) (MWRR) guarantees them. We then show that under general preferences, MWRR may guarantee some notions while failing others. Finally, through experiments, we show that MWRR may still perform well even for adaptations it does not formally guarantee.

Multi-Winner Voting with Argumentative Ballots cs.GT

We introduce multi-winner voting with argumentative ballots (MVArg) and investigate theoretical properties. As our conceptual contribution, we generalise approval ballots to argumentative ballots, thereby allowing voters to express defeasible preferences over candidates. We accordingly generalise voter cohesion and justified representation axioms JR, PJR and EJR. As our theoretical contribution, we establish several key results. First, MVArg is strictly more expressive than multi-winner voting with approval ballots (MV). Second, our notions of cohesion and justified representation are conservative generalisations of their counterparts in MV. Third, the MVArg counterpart of JR can always be satisfied, whereas the counterparts of PJR and EJR cannot always be. Fourth, although verifying whether a winner set satisfies the MVArg counterpart of JR is already coNP-hard, such a winner set can be constructed in polynomial time. All definitions, propositions, auxiliary lemmas and theorems have been formalised and mechanically checked in Lean 4.

Learning to Price with Persuasion cs.GT

Motivated by modern marketplaces, where the platform or the seller routinely gathers detailed user profiles, we study a novel learning theoretic model that simultaneously involves information and mechanism design. Specifically, we consider the economic setting recently introduced by Bergemann et al. (2022), where in addition to the menu of quality-price pairs, the seller offers information on the value of the match between product quality and buyer's taste via a signaling scheme. We relax the assumption that the seller knows the buyers' belief about the distribution of tastes and study the sample requirements of designing a revenue maximizing scheme. We consider both the batch setting where we have access to data from a set of i.i.d. buyers and an online demand query model where we observe the buyers' behaviors to seller's schemes. Despite the apparent non-convexity of the problem, we also give the first FPTAS to compute a scheme that maximizes the revenue within an arbitrarily small additive loss, which was left open by Bergemann et al. (2022). Overall, this brings a new learning perspective in asymmetric economic settings where buyers and sellers know different types of information.

Error-Aware Reverse Auction Mechanism for Large Language Model Routing cs.GT

Routing each query to a cost-effective large language model (LLM) is critical for balancing quality and cost, yet most routers rely on a centralized task center to predict model performance, creating an information-risk mismatch and a scalability bottleneck as the model pool grows. We propose a market-based routing paradigm that shifts ex-ante prediction to LLM providers via a reverse auction, where providers bid with self-predicted success probabilities and execution costs. To account for inherently noisy provider predictions and center evaluations, we introduce the \textit{\textbf{E}rror-\textbf{A}ware \textbf{R}everse \textbf{A}uction \textbf{M}echanism} (EA-RAM), which explicitly models this inherent Dual Error. We prove that EA-RAM is Bayesian incentive compatible and individually rational under the Dual Error, establish sufficient conditions for center rationality, and derive an explicit welfare-loss bound. We further identify robustness effects: opposite-signed errors can cancel, vanishing-tail link functions (e.g., logistic) stabilize clear-cut cases via saturation, and extra noise smooths belief maps, reducing the gains from marginal manipulation. Experiments on simulations and real-world benchmarks show that EA-RAM is robust to the Dual Error and achieves a better cost--performance Pareto frontier than centralized baselines, with additional gains when providers contribute local information, validating its practical effectiveness.

BenchmarksFull tables
Artificial AnalysisIntelligence Index

Composite score across coding, math, and reasoning

#ModelScoretok/s$/1M
1GPT-5.560.264$11.25
2Claude Opus 4.757.348$10.94
3Gemini 3.1 Pro Preview57.2138$4.50
4GPT-5.456.881$5.63
5Qwen3.7 Max56.60$0.00
SWE-rebench

Agentic coding on real-world software engineering tasks

#ModelScore
1Claude Opus 4.665.3%
2gpt-5.2-2025-12-11-medium64.4%
3GLM-562.8%
4Junie62.8%
5gpt-5.4-2026-03-05-medium62.8%