The Inference Report

The GitHub trending set reveals a sharp bifurcation between two developer priorities: making AI agents actually useful in constrained environments, and building the infrastructure those agents need to reason effectively.

One cluster tackles the hard problem of running large language models where they don't naturally fit. AirLLM runs 70B inference on a 4GB GPU through memory optimization. KTransformers provides a framework for heterogeneous inference across different hardware. CUA and jcode treat the agent itself as a harness to be scaled and benchmarked rather than a black box to be prompted. These aren't about making models faster in the abstract; they're about fitting production workloads into real hardware budgets. The practical constraint is explicit: developers are shipping agents that need to work offline, on limited devices, or across fleets where cloud inference costs become prohibitive.

The second cluster solves the information problem: agents hallucinate or make poor decisions when they lack context. Code-review-graph builds a persistent, queryable map of a codebase so AI tools only retrieve what's relevant rather than drowning in irrelevant files. Wigolo adds local-first web search and research without API dependencies. WrenAI layers governance and context over text-to-SQL so agents generate trustworthy queries against real data sources. GitHub's Copilot SDK and PostHog's MCP integration point to a shift away from generic chat interfaces toward agent tooling embedded in existing workflows. The pattern is clear: developers are moving past "give the AI access to everything" toward "give the AI access to the right things, verifiably." The discovery repos on data quality (FiftyOne) and time series modeling (GluonTS) suggest this infrastructure work extends to training and evaluation, not just inference. Trending repos like codecrafters-io/build-your-own-x and the various learning frameworks show that building agents is becoming a taught skill, not a novelty. The investment is in making agents reliable enough to ship, not just impressive enough to demo.

Jack Ridley