For years, “AI on your phone” really meant “your phone sending data to a server and showing you the answer.” That’s changing. Every major chipmaker — Apple, Qualcomm, Google, MediaTek — now ships a dedicated neural processing unit (NPU) alongside the CPU and GPU, purpose-built to run machine learning models directly on the device.

Why run AI locally at all?

Three reasons keep coming up:

• Latency. A round trip to a cloud server adds noticeable delay. For features like live translation, transcription, or camera processing, that lag is the difference between “useful” and “annoying.”
• Privacy. Keeping voice, photo, and message data on-device means less of it ever leaves your phone. That’s a meaningful selling point as consumers get more wary of how their data is used.
• Cost. Running inference on millions of phones is a lot cheaper for a company than running it on millions of cloud requests. On-device AI shifts the compute bill from the company’s data center to hardware the customer already bought.

What’s actually running locally

Small, quantized versions of language models now fit comfortably in a few gigabytes and run fast enough on modern NPUs for tasks like summarizing notifications, suggesting replies, or transcribing voice memos in real time. Larger, more capable requests — the kind that need a full-size model — still get routed to the cloud. Most phone AI today is a hybrid: a fast local model handles the easy cases, and a cloud model handles the hard ones.

The catch

On-device models are smaller and less capable than their cloud counterparts, so quality is a real tradeoff. There’s also a fragmentation problem — an app optimized for one chipset’s NPU doesn’t automatically run well on another, which is part of why cross-platform on-device AI tooling has become its own competitive battleground between Apple, Google, and Qualcomm.

Expect the split to keep shifting toward the device as NPUs get more powerful with each hardware generation. The cloud isn’t going away, but “AI phone” is increasingly a hardware claim, not just a software one.