The Inference Report

The GitHub ecosystem is splitting into two distinct waves of infrastructure investment. The first wave, now mature, addresses the mechanics of AI deployment: how to run models efficiently on constrained hardware, how to orchestrate them into working systems, how to move data through pipelines. LanceDB handles multimodal retrieval at the application layer. OpenVINO optimizes inference across hardware targets. Haystack provides explicit control over retrieval, routing, and memory in LLM pipelines rather than hiding these concerns behind abstraction. These repos solve specific problems in the production path from model to user.

The second wave, still accelerating, treats AI agents as a primitive worth building on top of. Instead of asking how to run a model, developers are now asking what you can build when agents become composable. Harness designs domain-specific agent teams and generates their skills. Design.md gives agents a structured understanding of design systems so they can reason about visual identity. OpenMontage chains 52 tools into 500+ agent skills, turning video production into something an agent can orchestrate. LobsterAI runs on desktop and accepts commands from messaging apps, treating the agent as infrastructure for getting work done across multiple surfaces. This isn't about better inference or faster training. It's about treating agents as building blocks that can be specialized, composed, and deployed across different contexts. The star counts on agent frameworks like Hermes and the sustained attention to repos like Orca suggest this layer is where developer effort is concentrating. The practical question has shifted from "can we run this model" to "what can we build if we treat the agent as a first-class abstraction."

Jack Ridley