What is On-device AI?

On-device AI, also called local AI or edge AI, is when an AI model runs on the hardware in your hand or on your desk rather than on a company's servers. When you ask a question, the inference happens on your own chip, and your prompt never has to leave the device.

How it works

Someone has to put a copy of the model's weights on your device, either as part of an operating system or as a download you choose. Your device's processor, usually a GPU or a dedicated neural engine, then runs the model. Because phones and laptops have far less memory and power than data centers, on-device models tend to be smaller, or compressed with quantization so they fit. Many are open-weight models that anyone can download.

Benefits

• Privacy. Your text, files and images stay on your machine, which matters for sensitive work or for data you are not allowed to share.
• Works offline. No internet connection needed.
• No per-request fee. After you have the hardware, each question costs only electricity.
• Low delay. There is no network round trip.

Limits

• Smaller models. In general, the most capable models are too large for consumer hardware, so on-device answers can be weaker on hard tasks.
• Memory is the ceiling. The model and its working data must fit in your device's memory. See unified memory and VRAM.
• Speed varies. A model can fit and still run slowly. Check tokens per second.
• You manage it. Updates, safety filters and fixes are your job, or your device maker's.

Hybrid setups

Many products mix both. A small local model handles quick, private tasks, and a harder question goes to a larger cloud model. Deciding which one gets a request is a form of model routing. If you use a hybrid app, check what it sends to the cloud and when, because "on-device" in marketing does not always mean everything stays local.

When to choose it

Pick on-device AI when privacy, offline use or steady cost matters more than top-tier ability. Pick the cloud when you need the strongest model available. Many people end up using both.