The Vision
Cloud AI APIs are powerful, but they come with costs: latency, privacy concerns, rate limits, and dependency on external services. Locailly is an experiment in a different model — one person operating like a coordinated team, entirely on local hardware.
The goal isn't to replace cloud models with worse local alternatives. It's to decompose work into small, specialized tasks that small local models can handle well, then orchestrate those models into a coherent workflow. The result is a system where one person can manage multiple parallel workstreams without needing a team, a cloud budget, or a privacy compromise.
Architecture
Locailly is built around several core components that work together:
Core Components
- Local Brain: The orchestrator that decides which model handles which task, manages state, and coordinates the workflow
- Task Tracks: Parallel workstreams that maintain their own context and progress — like having multiple browser tabs, but for AI-driven work
- AI Pit Crew: Specialized small models, each optimized for a specific job — classification, summarization, formatting, validation
- Capability Routing: A system that matches incoming tasks to the best-fit model based on task type, complexity, and available resources
Worker Roles and Validation
Each model in the pit crew has a defined role with clear input/output contracts. The Triage worker classifies incoming tasks by type and urgency. The Summarizer condenses long text into actionable points. The Formatter structures output for different downstream consumers. The Checker validates that outputs meet defined criteria before they're passed along.
Validation runs at every handoff point. When one worker's output becomes another's input, the system checks format, completeness, and basic quality signals. This prevents cascading errors — a malformed summary doesn't make it to the Formatter, and a bad format doesn't reach the output stage.
NearbyNodes: Optional Distribution
For setups with multiple machines on the same network, NearbyNodes adds optional distribution. Work can be routed to other local machines based on capability and load. A Raspberry Pi might handle lightweight classification while a workstation runs the heavier models. This is opt-in — Locailly works fine on a single machine.
The distribution layer is designed to be invisible when not needed. If there's only one machine, NearbyNodes simply isn't active. No configuration, no overhead, no complexity unless you want it.
What's Working
For structured tasks with clear rules, the local orchestration model holds up well. No network latency, no rate limits, no privacy concerns. The pit crew approach makes iteration easier — swap out one model without affecting the others. It's modular by design.
The Local Brain's routing logic is the most interesting piece. Getting it to correctly match tasks to the right model — and to know when no local model is sufficient — turns out to be the core challenge. Too aggressive in routing to local models and quality drops. Too conservative and you're back to cloud dependency.
Current Limitations
This is an active experiment, not a shipped product. The models are small and limited in capability. They work well for structured tasks but struggle with anything requiring deep understanding or creativity. Resource usage is a real constraint — running multiple models simultaneously requires meaningful compute, which limits practical applications on consumer hardware.
The most honest assessment: Locailly works for specific, well-defined workflows. It's not a general-purpose AI replacement. It's a tool for people who know exactly what they want automated and can define the task boundaries clearly.
Status: Active Experiment (WIP)
Locailly is an active experiment, not a shipped product. The concepts are being explored and tested — Local Brain routing, Task Tracks lifecycle, Pit Crew model selection — but there's no public release planned yet. This is work-in-progress.