The Inference Report

The GitHub trends this week show two distinct movements: developers building practical infrastructure for LLM deployment, and a surge in tools that let anyone run capable AI models locally. The infrastructure play is clear across repos like Microsoft's agent-framework and block/goose, which abstract away the complexity of orchestrating multi-step AI workflows. These aren't thin wrappers around API calls; they handle state management, tool execution, and error recovery across different LLM backends. onyx-dot-app/onyx takes a different angle, packaging chat and RAG capabilities into a self-hosted platform that works with any LLM, positioning itself as the open alternative to proprietary AI platforms. What connects these is a shared bet that the bottleneck isn't model quality anymore but infrastructure maturity.

The local-first movement is equally pronounced. MLX-VLM lets you fine-tune vision language models on Mac hardware without cloud dependencies. AutoRAG focuses on the unglamorous but necessary work of evaluating and optimizing RAG pipelines, suggesting teams are past the honeymoon phase and now optimizing production systems. dstack addresses a real friction point: provisioning GPU compute across fragmented hardware ecosystems without vendor lock-in. Meanwhile, siddharthvaddem/openscreen strips away the complexity from demo creation, eliminating subscription friction that keeps people in proprietary tools. The pattern here isn't about raw capability but about control and cost. Developers are voting with their forks for systems they can run, modify, and own, particularly where cloud pricing or vendor lock-in created friction. The discovery repos reinforce this: defradb's peer-to-peer database model, AutoRAG's focus on evaluation rigor, and dstack's multi-cloud provisioning all solve real operational problems that emerge only after you've deployed something to production.

Jack Ridley