Field report · · The 408
How Fred answers in 1.66 seconds: the model is the last thing a request reaches
Fred runs a real house on local models. The trick is that the model is the last thing a request reaches, not the first. Most requests never touch a GPU, and the ones that do get a model sized for the job. If you’re running a home assistant on one consumer GPU, this is the layout that kept voice fast and the house safe.
One house, one 12 GB card and a Mac mini, measured in September and October 2026. Not a benchmark.
The path a request takes
Fall through the cheap rungs first
- 1
The hub’s own intents No model
“Turn off the kitchen lights” is matched by Home Assistant’s built-in sentence intents and done locally. The voice pipeline tries these before anything else.
Speech-to-text: faster-whisper on the GPU - 2
Plain-command parser No model
House-specific phrasing like “goodnight” resolves to an action or a reading in Python. Still no model.
- 3
Intent router No model
Classifies what’s left and picks a small toolset: house, audio, energy, vehicle or tasks. Write tools are offered only when the request asks for an action, so a question can never cause a change.
- 4
Fred Fast: 9B, always resident Model
Qwen3.5 9B at Q4_K_M, 40K context, with vision. It lives fully in VRAM on a 12 GB RTX 4070 and is never evicted. It handles the unmatched phrasing, the tool calls and the wording.
Ollama · RTX 4070 12 GB · about 7.6 GB resident - 5
The brain: 35B mixture of experts, on a different machine Model
Qwen3.6-35B-A3B, with 3B parameters active per token, handles deeper questions, background agents and long summaries. It runs on a Mac mini so it never fights Fast for the GPU.
llama.cpp · M4 Pro, 48 GB unified memory · two 64K slots - ⛨
The guard Code
Every tool call is rated by risk. Risky ones need a single-use approval that expires in five minutes. Reversible changes snapshot the old state first, so “undo” works.
Measured, not guessed
Why the 27B didn’t stay on the GPU
A dense 27B at Q4 doesn’t fit in 12 GB, so it spilled into system RAM. It worked, but every token waited on the CPU half. It became an opt-in mode, then was retired when the mixture-of-experts brain went live on the Mac.
Speed wasn’t the real problem. Eviction was. When the big model loaded on the same card, it pushed the small one out, and voice stalled behind it:
Fast resident, nothing else competing.
Fast evicted and reloaded. One model per card for anything latency-sensitive.
Learned the hard way
Rules the code enforces
- Verdicts live in code; the model only words them.“Is the door locked?” is answered from the lock’s state, and the model phrases the answer. An early version let the model decide, and it said “unlocked” for a locked door.
- A model offered a write tool will use it unasked.During a planning question, the bigger model took a real snapshot of a virtual machine. Now tools that change things appear only when the request is an action.
- A model with no tool doesn’t go quiet. It makes something up.Asked about room tracking with no presence tool in scope, it confidently said the house had none. The fix was routing and grounding, not a bigger model.
- Size the model to the rung.A 9B for the hot path, the big one for thinking, and nothing that controls the house waits on either.
If you have one 16 GB card
What I’d do with one GPU
- Keep a 9B to 14B dense model fully in VRAM for the hot path, with room for speech-to-text beside it. Never let a bigger model evict it.
- For the thinking tier, try a mixture-of-experts model like a 30B–35B-A3B before a dense 27B. Only about 3B parameters are active per token, so pushing the expert weights to system RAM costs far less than spilling a dense model. In llama.cpp that’s
--n-cpu-moe N(or-ot "exps=CPU"). I haven’t measured this on a 16 GB card, so treat it as the next thing to try. - Put deterministic paths in front of the model. Every request handled by intents or plain code is one the GPU never sees.
Which of your requests could skip the model entirely?
Numbers are from Fred’s own logs and benchmarks, September and October 2026. Hardware: an RTX 4070 12 GB passed through to a virtual machine, and a Mac mini M4 Pro with 48 GB.