Running a 20B MoE Model at 120 Tokens Per Second on an iPhone

Running a capable AI model on a phone used to mean accepting severe limitations. Small models, slow speeds, and battery drain defined the experience. That story changed dramatically in 2026. A new generation of on-device models, led by innovations like the Maple-Preview MoE architecture, now delivers performance that seemed impossible not long ago.

Specifically, running a 20 billion parameter mixture-of-experts model at over 100 tokens per second on an iPhone. Let that sink in. Phone-grade hardware, laptop-class AI speed.

What Makes This Possible

Two key innovations converged to make this happen.

Mixture of Experts (MoE): Traditional models activate all their parameters for every token. MoE models activate only a fraction at any given time. A 20B MoE model might use just 2-4 billion parameters per token. That means much less computation per step while retaining the quality benefits of a larger total parameter count.

Mobile-optimized inference engines: Apple's Core ML framework and third-party engines like MediaPipe and MLC-LLM got dramatically better at using phone hardware efficiently. They leverage the Neural Engine, GPU, and unified memory architecture to maximize throughput.

The combination of efficient model architecture and optimized runtime produces speeds that feel genuinely fast in practice.

Real-World Performance

At 120 tokens per second, responses appear almost instantly. Ask a question, and the answer finishes before you finish reading the prompt. That speed transforms what on-device AI can do.

Real-time translation during conversation? Smooth. Writing assistance that keeps up with typing? No lag. Analyzing a document and summarizing it? Takes seconds.

The quality holds up too. Modern small models punch far above their weight. The 20B MoE model handles writing, coding assistance, analysis, and general reasoning at a level comparable to much larger models from just a year ago.

What This Means for Privacy

On-device inference means data never leaves the phone. For sensitive applications, that matters enormously. Health queries, personal finance questions, private conversations: all processed locally without any cloud dependency.

Companies building health, finance, and productivity apps are paying close attention. The ability to offer AI features without sending user data to servers solves a major compliance and trust challenge.

Battery Impact

Running AI on a phone traditionally meant watching the battery percentage drop. The new generation of efficient models and inference engines reduced that impact significantly.

Short interactions cost minimal battery. Extended sessions do drain faster, but not catastrophically. For most use cases, the battery trade-off is acceptable given the capability gained.

The Road Ahead

Phone-based AI will only get better. Model architectures continue shrinking while maintaining quality. Inference engines get more efficient with each generation of mobile chips.

By late 2026, expect flagship phones to handle models in the 50-100B parameter range at useful speeds. The line between "phone AI" and "cloud AI" will blur further.

The future of AI is not just in massive data centers. It is in pockets and purses, running fast, running private, and running right now.