What local AI actually means, and when it beats the cloud

“Local AI” gets used to mean three or four different things, usually by people who would rather you did not look too closely. So let us be precise.

Local AI means the model weights sit on your disk, the maths runs on your chip, and the answer is produced without a network request. That is the whole definition. Everything else follows from it.

The test is simple: turn off Wi-Fi. If the feature still works, it is local. If it spins, it was never local, whatever the marketing page says.

The three things people call “local” that are not

“Your data is encrypted in transit.” So is everything. That is TLS, and it has been table stakes since the nineties. It tells you nobody read your data on the way to the server. It tells you nothing about what happened once it arrived.

“We don’t train on your data.” Better, and worth having, but it is a promise rather than an architecture. It can be changed in a policy update. It does not survive the company being acquired, subpoenaed, or breached.

“Processing happens on a private cloud.” This is the slipperiest one, because it is technically impressive and still not local. Your data left your machine. It went somewhere you cannot inspect. The security of that arrangement rests on the operator doing exactly what they said, forever.

Local AI is a different kind of claim. It is not a promise about behaviour, it is a fact about where the computation happened. There is no request to log because there is no request.

Why this became possible

Two things changed at roughly the same time.

Models got dramatically more capable per gigabyte. A model you can fit in a few gigabytes today does work that needed a data centre three years ago. Quantisation is a big part of this: storing the model’s numbers at lower precision shrinks it a lot while costing surprisingly little quality.

And Apple Silicon turned out to be unusually good at this specific job. On a traditional PC, the CPU and GPU have separate pools of memory, so running a model means shuttling gigabytes back and forth across a bus. M-series chips share one pool. The GPU reads the model where it already sits. That is why a fanless laptop can do work that needs a noisy tower elsewhere.

What local AI is genuinely good at

Small models are not small versions of big models. They have a different shape of competence, and matching the work to the shape is the whole game.

Anything with a tight, well-specified job. Finish this sentence. Is this message urgent or not. Pull the invoice number, date, and total out of this receipt. Rewrite this paragraph to be shorter. These are bounded tasks with a clear right answer, and a small model does them reliably.

Anything that needs to be fast. A local model answers in tens of milliseconds because there is no round trip. That sounds like a minor detail until you try to build autocomplete on a cloud call, at which point the latency makes the feature unusable. Some things are only possible locally, not merely cheaper.

Anything ambient. If something watches your screen all day, or listens through every call, the cloud version is both a privacy problem and a bill. Running it locally makes it free at the margin, which changes what you are willing to leave switched on.

Anything sensitive. Your medical notes, your salary negotiation, your unfinished resignation letter, your therapy journal. The reason to keep these local is not that cloud providers are villains. It is that the safest place for data nobody needs to see is a machine nobody else can reach.

What it is honestly worse at

Long, open-ended reasoning. Synthesising forty documents into a strategy memo. Writing a lot of correct code in one go. Obscure factual recall.

This is not a gap that will close by next quarter. A model small enough to sit on your laptop has fewer parameters than one occupying a rack, and for genuinely hard reasoning that difference shows.

The useful conclusion is not “local AI is not ready”. It is that the two are for different work, and a tool that pretends otherwise is lying to you in one direction or the other. The ambient, constant, personal layer should be local. The occasional heavy lift can reach for something bigger, as long as it is your choice and you can see it happening.

What changes when the model is already on your machine

Here is the part that is easy to miss. Once a capable model runs locally for free, the economics of software invert.

Cloud AI features are metered. Every suggestion costs the vendor something, so they are rationed: a credit system, a monthly cap, a “fair use” clause. The product is shaped around making you spend calls carefully.

A local model costs nothing per call. So it can run constantly, on everything, without anyone counting. It can watch a folder all day. It can check every incoming message. It can offer a completion on every sentence you type and throw away the ones you ignore. Being wrong is cheap, which means it can afford to try.

That is the real unlock, and it is why local AI is more interesting as a foundation than as a feature. It is not “the same thing, but private”. It is a different set of things being possible at all.

How to tell if something is actually local

Four checks, in order of how hard they are to fake:

1. Turn off Wi-Fi. The honest test. Does the feature still work?
2. Watch the network. Little Snitch or macOS’s own Activity Monitor will show you whether traffic leaves during the operation.
3. Look for the model on disk. Local AI needs weights. They are large and they have to live somewhere. If nothing multi-gigabyte was ever downloaded, nothing is running locally.
4. Read what the privacy policy claims about processing, not about training or encryption. Those are different questions and vendors often answer the easy one.

Where Halo sits

Halo runs its ambient layer on your Mac: the autocomplete, the screen memory, the call transcription, the classification that decides whether something is worth interrupting you for. That work is constant, so it has to be free and fast, and it touches everything you do, so it has to be private. Local is the only architecture that satisfies all three.

For the occasional heavy lift, there is an optional cloud plan. It is off by default, it is a deliberate switch, and when it is off no ambient data has anywhere to go. The point is not that the cloud is forbidden. It is that reaching for it should be a decision you make, not a default you inherit.

See what runs on your Mac, or read the privacy contract if you want the specifics of what is stored and for how long.