My house has two AI brains. The lights don’t need either one.

The 1.66-second problem

I wanted Fred to feel like part of the house, not another computer I had to operate.

That meant voice control needed to be quick enough to feel natural. More importantly, it needed to be predictable.

There’s nothing particularly impressive about a voice assistant that answers instantly nine times and takes 30 seconds on the tenth. The tenth is the one you remember, especially when you’re standing in a dark kitchen wondering whether it heard you.

So I started measuring everything.

During the week of October 8–15, 2026, Fred’s route log recorded 5,879 turns. For the 5,613 voice responses that came in while nothing else was competing for his graphics card, the median response time was 1.66 seconds. Ninety-five percent finished within 3 seconds.

In plain English, half were faster than 1.66 seconds, and only the slowest 5 percent took longer than roughly 3 seconds.

These measurements start when Fred receives the words of a request and stop when he replies. Some entries also came from dashboard requests for short responses, rather than someone physically speaking. They aren’t a complete measure of how long every spoken interaction takes.

Still, they gave me something useful: evidence that the system could respond quickly and consistently when its resources weren’t being pulled in different directions.

The interesting part was how we got those numbers.

It wasn’t by finding the biggest AI model or buying more processing power. In fact, one of my earlier attempts to make Fred smarter produced the opposite result.

The improvement came from reducing how often he needed to use AI in the first place.

Start with the things that don’t need intelligence

Fred uses Home Assistant to manage the house. If you’re unfamiliar with it, Home Assistant is a platform that connects smart-home devices and lets them operate together.

It handles lights, sensors, climate systems, automations, and a growing list of other things that would otherwise be scattered across separate apps.

It also understands ordinary voice commands.

That matters because a request like “turn off the kitchen lights” doesn’t require a language model to solve it. Home Assistant already knows what a kitchen is, which lights belong to it, and how to turn them off.

Fred’s first job is to let Home Assistant handle those requests.

Before any command reaches the main AI system, a local speech-to-text model converts my voice into words. That model runs on the graphics card inside my home server and uses about 2.1 GB of its 12 GB of memory.

The audio processing stays inside the house.

Once Fred has the words, Home Assistant gets the first opportunity to understand them. Its built-in commands can handle many common requests without involving a larger AI model.

That takes care of the obvious stuff. But every house develops its own vocabulary.

“Goodnight.”

“A little dimmer.”

“Undo that.”

These phrases might have a very specific meaning in one home and mean something entirely different in another.

For Fred, I use plain code to recognize many of them. A set of rules matches the request to an action, without asking a model to interpret it.

That sounds less impressive than artificial intelligence. It’s also much more reliable for commands that should behave exactly the same way every time.

I learned that lesson while trying to get Fred to restart a media server.

The smaller AI model kept responding with words instead of doing anything. Sometimes it insisted the server was already running. Other times it politely refused to perform an action it was supposed to be allowed to do.

I changed its instructions. I refined the wording. I tested different approaches.

The behavior kept returning.

Eventually, I took the command away from the model and wrote a direct rule to handle it.

Problem solved.

There was no breakthrough in artificial intelligence. The solution was to stop asking artificial intelligence to do a job that ordinary software could perform better.

I’ve since applied that principle throughout the house.

There are 156 automations in the system, and the essential operation of lights, locks, and alarms doesn’t depend on a language model being available.

If Fred’s AI goes offline, the house doesn’t suddenly forget how to be a house.

That’s a much more useful definition of reliability than whether an assistant can hold an impressive conversation.

The part where Fred actually has to think

Of course, not everything can be reduced to a predictable command.

Sometimes I ask a question. Sometimes I phrase something in a way Home Assistant doesn’t recognize. And sometimes I want Fred to investigate a problem rather than simply operate a device.

That’s where the AI earns its keep.

Fred has access to more than a hundred software tools. Some can retrieve information about the house. Others interact with audio systems, energy monitoring, tasks, infrastructure, and connected devices.

A tool is essentially an action or information source made available to the model. It might read a thermostat, check a server, or operate a light.

Giving an AI model access to all those tools at once sounds convenient.

In practice, it creates problems.

Smaller models can struggle when they’re asked to choose from an enormous list of possible actions. They may select the wrong tool, overlook an obvious one, or generate an answer instead of performing the requested action.

So Fred has a router.

Before a request reaches the AI model, ordinary code attempts to determine what kind of request it is. If I’m asking about the house, Fred receives tools relevant to the house. If the question concerns audio, the available tools change.

The model gets a smaller, more relevant selection instead of the entire toolbox.

There’s also an important distinction between asking a question and requesting an action.

If I ask, “Should the heat be on?” Fred can investigate and answer. That question should not give him permission to change the thermostat.

Tools that modify the house are made available only when the request actually calls for an action.

That distinction exists because I learned what happens when you don’t enforce it.

During one test, the larger model was supposed to help with planning. Instead, it used an available tool to create a real snapshot of a virtual machine.

Nobody asked for the snapshot.

The model simply had a tool available and decided to use it.

It wasn’t a catastrophic failure, but it was an excellent demonstration of why permissions shouldn’t depend entirely on an AI model interpreting someone’s intentions correctly.

The router helps prevent that. It isn’t perfect, though.

In a test using the phrases it had been designed to recognize, it selected the correct tool area 96.4 percent of the time.

I then tried 88 alternate phrasings specifically designed to avoid the expected keywords.

The result dropped to 13.6 percent.

That’s a fairly spectacular decline, and a useful reminder that keyword matching isn’t intelligence.

The router reduces unnecessary work. It cannot be trusted as the final authority.

Fred needs other ways to recover when a request reaches the wrong tools, and independent safeguards that don’t depend on the router making the right decision.

One of those ways now reaches outside the house. When the keyword router finds nothing at all, the words go to a cloud model that can suggest one more area of tools. It can’t turn a question into permission to act, and anything with personal or sensitive words never leaves: the Mac mini answers those instead.

Which brings us to the model itself.

The small brain does most of the talking

Fred’s everyday model is Qwen3.5 9B.

The “9B” means it has approximately nine billion learned parameters. You can think of parameters as the values that help a model recognize patterns and generate responses.

More parameters generally require more memory, although model size alone doesn’t determine whether a system will perform well.

Fred’s model is compressed so it can run locally on the graphics card. It occupies about 7.4 GB of memory.

Combined with the speech-to-text model, that’s approximately 9.5 GB of a 12 GB card before accounting for the rest of the workload.

Memory is tight, but it fits.

The reason I selected this particular model wasn’t that it won a leaderboard or had the most impressive specifications.

It passed the tests that mattered in my house.

In one tool-selection test, it made the correct choice 12 out of 12 times. It could acknowledge when information wasn’t available instead of making up an answer. And it could handle a longer conversation without exceeding the card’s memory.

That last part is important.

If a model doesn’t fit in the available graphics memory, some of its work can spill into ordinary computer memory. Depending on the hardware, the performance difference can be enormous.

I measured it.

A 14B model that fit on the card produced 43.9 tokens per second. A larger 27B model that didn’t fit dropped to 8.5 tokens per second.

After some optimization, the 27B model improved to 16.7 tokens per second. Better, but still nowhere near the smaller model.

A token is a piece of text that an AI model processes. The rate tells you how quickly it generates a response once it starts writing.

The larger model had more capacity, but its size made it a poor choice for something I wanted to talk to throughout the day.

There was another problem.

Models don’t always remain in memory indefinitely. In one week, Fred’s smaller model unloaded after 30 minutes of inactivity, despite the design goal of keeping it available. Eight requests paid roughly six extra seconds while it loaded again.

Six seconds doesn’t sound terrible until you’re waiting for an assistant that normally answers in less than two.

It was another reason to keep the everyday model small and, where the runtime allows it, resident in memory.

But I still wanted Fred to handle more difficult questions.

And for a while, I thought I could get the best of both worlds by giving him a second, larger model on the same graphics card.

That was a mistake.

I gave Fred a bigger brain. He got slower.

The larger model was supposed to handle the difficult work: complex questions, longer summaries, planning, and problems that require more reasoning.

The small model would continue taking care of everyday interactions.

On paper, it made sense.

In reality, both models were competing for the same memory.

When the larger model loaded, it displaced the smaller one. The next voice request could then wait while the system shuffled models around instead of simply answering.

I could see the effect in the logs.

During the October 8–15 measurement window, 5,613 responses recorded when the smaller model had the card to itself had a median time of 1.66 seconds and a 95th-percentile time of 3 seconds.

For 40 responses recorded while the larger model occupied the card, the median rose to 3.1 seconds.

The 95th percentile reached 40.5 seconds.

These aren’t equally sized samples or a controlled laboratory comparison. They’re observations from the same house during the same week.

But the difference was large enough to expose a very practical problem.

Fred could be fast, and Fred could use a bigger model, but sharing that graphics card was making the experience unpredictable.

So I moved the bigger brain to another computer.

Since October 24, Fred’s larger brain has been running on a Mac mini with 48 GB of unified memory.

It’s now a Qwen3.6-35B-A3B model, using a design called mixture of experts. Although it has approximately 35 billion parameters, only about three billion are active when generating each token.

That allows a model with substantial overall capacity to use a smaller portion of its parameters for each step.

In a test on the Mac mini, it generated 42.5 tokens per second. That’s comparable to the 43.9 tokens per second I measured from the smaller 14B model that fit on the graphics card.

There’s a catch, and it’s a big one.

The larger model can generate text fairly quickly once it starts. Reading a large amount of information before it begins answering is another matter entirely.

I measured that too.

These tests were performed on October 24 while the Mac was also doing other work.

Hand the model a few pages of information, and it might take several seconds before responding. Hand it a short book, and the delay can exceed a minute.

That’s acceptable when I’m asking Fred to investigate a complicated problem, examine a history of events, or produce a detailed report.

It’s ridiculous when I’m asking him to turn off a lamp.

Now the larger brain can work without displacing the smaller model from its graphics card. The two machines have different responsibilities, and a slow reasoning task doesn’t need to hold up routine interactions.

I should be clear about one limitation: I don’t yet have a full week of comparable voice-performance data from after the move.

The 40.5-second result is evidence of the old problem. It isn’t proof that every delay has been eliminated.

That distinction matters, particularly when you’re writing about your own system. It’s easy to confuse fixing the architecture with proving the outcome.

The next set of measurements needs to show whether Fred stays responsive while the larger model is busy.

The problem with letting an AI operate your house

Speed is only part of the story.

An assistant that answers quickly but does the wrong thing isn’t particularly useful. An assistant that confidently claims to have done something when nothing happened is worse.

And once you give an AI the ability to operate physical devices, mistakes are no longer confined to a chat window.

I don’t want Fred deciding to unlock a door because he misunderstood a sentence. I don’t want a question about the heating system to become an instruction to change it.

I also don’t want an AI model determining its own safety rules.

So Fred has a separate guard.

It’s ordinary software that checks actions before they happen. The guard evaluates the requested operation and the relevant device state, rather than trusting the model’s explanation of why the action is reasonable.

Different actions receive different levels of scrutiny.

Reading the status of a porch light is low consequence. Turning on a few lights can generally happen without interrupting me. Changing a large number of devices may require confirmation. Security-sensitive actions require stronger authorization.

Some actions cannot be approved by voice at all.

Where it’s technically possible, Fred also records the previous state before making a reversible change. That gives the system a better chance of supporting an actual “undo” rather than merely apologizing.

The point isn’t to make every request difficult. It’s to make the ordinary requests easy while keeping consequential decisions under human control.

For this to work, the rules must exist outside the model.

A language model can misunderstand instructions, invent explanations, or respond differently to an unfamiliar phrasing. The independent guard shouldn’t change its decision because the model found a more persuasive way to describe the same action.

But even a correctly authorized action can fail.

A device might be offline. A command might time out. A service might report success before the actual device has changed.

Fred taught me that lesson in a particularly annoying way.

I asked him to dim a light.

He replied, “It is a little dimmer.”

It wasn’t.

The light hadn’t changed at all.

When I checked what happened, there had been zero tool calls. Fred hadn’t even attempted to adjust the light.

He had simply generated a convincing sentence.

From the perspective of a language model, that’s understandable. It had seen a conversation, recognized the kind of answer that usually follows such a request, and generated one.

From the perspective of someone standing in a room with an undimmed lamp, it was useless.

I tried correcting him conversationally. That didn’t solve the underlying problem either. Sometimes he would apologize for an action he’d actually performed correctly.

I was arguing with a text generator about the physical state of my house.

There had to be a better way.

Now Fred has a verification step that checks a response against what the system actually did.

If he claims to have changed a light, the system needs evidence of an appropriate tool call and a state reading that supports the result.

If that evidence is missing, the response can’t simply present the action as completed. Fred can attempt a controlled recovery or explain that the change wasn’t confirmed.

The distinction seems small, but it changes how much I can trust the assistant.

“I’ve dimmed the light” is a claim.

“The lamp is at 40 percent” is a report that should be supported by a reading from the actual device.

I call the rule said is not done.

It applies to much more than lights.

A backup isn’t successful because a script exited without an error. An automation isn’t working because its service is running. And an AI assistant hasn’t completed a request simply because it says it has.

You need to check the result.

Some of my worst ideas sounded reasonable at the time

One thing I’ve learned from working on Fred is that a technically reasonable solution can still produce a terrible result.

The larger model sharing a graphics card was one example. I have several others.

At one point, Fred invented two tasks in a daily summary.

My response was to reduce how much freedom the language model had. I wrote code to construct the summary and let the model turn it into natural speech.

That sounded like a sensible way to prevent hallucinations.

It got worse.

Across ten test runs, the revised approach produced incorrect statements. Fred reported that nothing was overdue when three tasks were overdue. In another case, missing weather information turned into an invented description of clear skies and temperatures in the mid-70s.

It had actually been raining.

So I put the structure back.

In the small tests that followed, it got the summary right three times out of three, then four out of four.

The problem wasn’t that Fred had too much freedom to write sentences. It was that the information I gave him wasn’t sufficiently structured and authoritative.

I needed the software to establish the facts, then let the model explain those facts naturally.

Another time, Fred confidently informed me that the house didn’t have room-tracking capabilities.

It absolutely did.

The problem was that he didn’t have access to the tool he needed to check.

Instead of admitting that limitation, the model treated the absence of available information as evidence that the capability didn’t exist.

That exposed another bad assumption in the design.

A model that can’t find evidence doesn’t necessarily know that it hasn’t looked in the right place.

Now, when Fred can’t verify something about the house, the expected behavior is to say what he couldn’t check and why. The system can also record the missing tool access so I can address it later.

That’s considerably more useful than a confident answer based on nothing.

These failures might sound like small technical annoyances. Some were. But each exposed a design problem that could affect much more consequential interactions.

And they all pointed toward the same conclusion: giving a model more instructions isn’t always the answer.

Sometimes it needs better information. Sometimes it needs fewer permissions. Sometimes it needs a different tool.

And sometimes it shouldn’t be involved at all.

What the numbers actually tell me

I can demonstrate Fred doing something impressive in about 30 seconds.

I can ask a complicated question, show a dashboard updating, or have him explain an unusual event in the house.

Those demonstrations are fun. They don’t tell me very much about whether the system is reliable.

The useful evidence comes from leaving Fred running and measuring what happens when nobody is trying to make him look good.

During October 8–15, the gateway that distributes requests between Fred’s models recorded 6,132 requests to the local models.

Of those, 6,101 went to the smaller model.

Just 31 reached the larger one.

That’s roughly half of one percent.

The count doesn’t include commands handled entirely by ordinary code. Those never needed either model, so they don’t appear in the model gateway’s numbers.

That distinction is important. The numbers don’t tell me the total number of commands processed by the house, or the percentage of all household interactions that required AI.

They do show how little of the gateway’s recorded workload needed the larger model.

And I find that encouraging.

I’ve spent a considerable amount of time giving Fred more intelligence, more tools, more memory, and more ways to understand what happens around him.

Yet one of the best outcomes is that the most expensive part of the system doesn’t need to be involved very often.

The bigger brain isn’t supposed to be busy all day just to justify its existence.

It’s supposed to be available when the problem warrants it.

There’s still plenty to improve. The router’s performance against unfamiliar phrasing needs work. The smaller model’s memory residency needs to be more consistent. And I need a proper post-migration performance window to confirm the benefit of moving the larger model.

I also don’t consider the local test results a universal benchmark.

This is one house, with a particular collection of devices, one 12 GB graphics card, and a Mac mini. My workloads, hardware limits, and expectations won’t match everyone else’s.

But that’s part of why I think the results are useful.

These aren’t claims about what an AI system might accomplish in a carefully prepared demonstration.

They’re measurements from something I actually use.

If I were starting again, I’d leave the AI switched off

If you’re thinking about building an assistant for your own house, I wouldn’t start by choosing an AI model.

I’d start by figuring out what you want the house to do without one.

Get the lights, sensors, climate systems, and automations working reliably first. Make sure ordinary commands have predictable results.

Then identify the requests that need something more.

Maybe you want to ask why the heating system ran overnight. Perhaps you want a useful summary of unusual activity, or a natural way to operate several systems that weren’t designed to work together.

Those are good reasons to introduce AI.

But asking a model to decide which light switch to press every time you walk into a room seems like an expensive way to recreate a problem we’ve already solved.

I’d also spend more time on verification earlier in the project.

It’s tempting to focus on what an assistant can do. I spent plenty of time doing exactly that.

The more useful questions turned out to be what happens when it can’t do something, when it misunderstands a request, or when the system it depends on stops responding.

Those aren’t particularly glamorous questions. They’re the ones that determine whether people will trust the system enough to use it.

If I had to suggest one simple experiment, it would be this:

Turn off the AI model for an hour.

Use your house normally. Try the lights, media, climate controls, and everyday routines. Make a note of everything that stops working.

That’s your starting point.

Some things on the list genuinely need intelligence. Others probably shouldn’t have depended on it in the first place.

I still want Fred to become considerably smarter. There’s a lot more I want him to understand about the house, how its systems relate to one another, and how to identify problems before I notice them.

The new Mac mini gives him room to do more complex work without competing with everyday voice requests. Better context, improved memory, and more reliable reasoning should make him a more capable assistant.

But the goal isn’t to make every part of The 408 depend on Fred thinking.

It’s to make the house more responsive, dependable, and easier to live in.

After all the models, benchmarks, failed experiments, and occasional arguments with a light switch, I’ve become fairly convinced of one thing.

The smartest thing Fred can sometimes do is leave the thinking to something else.

And maybe, eventually, he’ll find my socks.