On-Device AI on Android: 2026 Build Guide

Why Fora Soft wrote this 2026 guide

Four years ago, shipping on-device AI on Android meant hand-rolling TensorFlow Lite builds, custom quantization scripts, and one vendor SDK per chip. In 2026 the stack is boring in the best way: LiteRT for the runtime, MediaPipe Tasks for vision and audio, and Gemini Nano through ML Kit for language. The catch is that most guides still online were written for the 2022 toolchain, and they will send you down build paths Google has since deprecated. Pick the wrong runtime and you burn a sprint migrating; miss the device-tier reality and your app freezes on the phones half your users actually own.

We’re Fora Soft — 250+ shipped projects since 2005, and an AI integration team that has put on-device inference into production. We built real-time video anomaly detection into NetCam, live proctoring with face landmarks into BrainCert, on-device OCR and speech into Memory Master, and edge inference into DSI Drones. This guide is the playbook we hand new engineers: which tools win, what breaks on low-end hardware, and when to let the cloud do the work.

Three promises on-device AI keeps that cloud cannot

1. Privacy holds by construction. Medical images, face biometrics, voice recordings, and behavioural telemetry never leave the phone. There’s no cloud sync to secure and no retention window to defend. For HIPAA, GDPR, and PCI-DSS work, on-device inference is compliance shorthand: the model runs inside the user’s sandbox, so there’s nothing to exfiltrate from your servers.

2. Latency is low and predictable. A cloud round-trip adds 50–300ms before the model even runs. On-device, a current flagship NPU runs SSD MobileNet object detection in 30–40ms, pose estimation in ~25ms, and Whisper-tiny speech recognition at 1.5–2× realtime. No congestion, no timeouts, no cold starts.

3. Offline is a feature, not a bug report. Airplane mode, tunnels, and rural dead zones don’t brick the app. Download the model once (typically 50–200MB for vision), and inference keeps working with no network. Users read that as fast and reliable, which is exactly what competitors leaning on cloud APIs can’t promise.

The cost angle: zero marginal inference

Cloud inference bills per call. On-device, the millionth inference costs the same as the first: nothing. You pay once to train and quantize the model, once in bandwidth to ship it, and after that the compute is the user’s device. We work the full numbers below, but the short version is that past a certain volume, cloud stops being an accounting rounding error and starts being a line item you have to defend.

Figure 1. The four-layer Android on-device AI stack: what your app calls at the top, what stores the model at the bottom.

The Android 2026 on-device AI stack

The stack splits into four layers: model formats, runtimes, hardware accelerators, and app-level APIs. Match the layer to the job and the tool choice mostly picks itself.

LiteRT: the runtime formerly known as TensorFlow Lite

Google renamed TensorFlow Lite to LiteRT in September 2024, and the project has moved well past a simple rebrand since. Your existing .tflite models still load unchanged, so migration is not a rewrite. What’s new is how you get top performance out of them.

The CompiledModel API is the part that matters

Two APIs, one recommendation. LiteRT 2.x keeps the classic Interpreter API (the old TFLite call surface) for backward compatibility, and adds the CompiledModel API as the modern path. CompiledModel is built to squeeze the hardware — it negotiates NPU, GPU, and CPU backends for you instead of leaving delegate wiring to hand-written boilerplate. For any new project, that’s the one to reach for.

On-device LLMs get their own runtime. LiteRT-LM is Google’s inference framework for language models on the edge. It runs Gemma-class models up to roughly 12B and reaches around 52 tokens/sec decode on the GPU backend on Android, with an OpenAI-compatible server in its CLI for local testing. If you’re shipping a chat or summarization feature and Gemini Nano isn’t available on the target device, this is your fallback.

Delegates are first-class now. Hexagon (Snapdragon) and GPU (Adreno, Mali) acceleration ships in the official release rather than as a vendor side-build. On a current flagship NPU, INT4 models run several times faster than on the ARM CPU alone.

MediaPipe Tasks: the library most teams should reach for first

MediaPipe Tasks is Google’s high-level layer over LiteRT and vendor NPUs. Instead of writing interpreter loops, you call ready-made tasks: HandLandmarker.detectAsync(frame), PoseLandmarker.detectAsync(frame), ObjectDetector.detect(frame). It handles quantization, delegate selection, frame resizing, and threading for you.

The vision tasks people actually ship

Hand landmarks. 21 keypoints per hand, model 4–8MB, ~10ms on a flagship NPU. We use it in BrainCert for hand-in-frame proctoring checks; it also underpins sign-language and gesture apps. Works offline with no extra setup.

Pose estimation. 33 body keypoints, 4–6MB (lite) or ~15MB (full), ~25ms on flagship. Fitness form-checks, physical therapy, motion capture, fall detection. Handles multiple people in frame.

Object detection (SSD MobileNet). ~90 COCO classes, 4–7MB, 30–50ms on a flagship NPU. This is the model behind the NetCam anomaly-detection case below.

Face detection and landmarks. Bounding box plus 468 facial landmarks, 5–10MB, ~15ms. Gaze and attention for proctoring, filters, landmark-based emotion cues.

Audio classification. Environmental sound classes (speech, music, applause, silence), 5–30MB, streaming on Android 10+. Accessibility context and voice-activity detection.

The LLM inference task

MediaPipe’s LLM Inference task lets you run a quantized model (Gemma-class INT4 and smaller) behind the same task interface, with context caching and token generation handled for you. It’s the bridge between the vision tasks above and the platform LLM below — one place to keep your inference code when a feature mixes vision and language.

Figure 2. Model size and latency drop sharply as precision falls. Accuracy cost climbs the same direction — measure it on your own data.

Gemini Nano and ML Kit GenAI: the platform LLM on Android

Short answer: on supported devices, you don’t ship an LLM at all — you call the one baked into the OS. Gemini Nano runs fully on-device through the AICore system service, and the recommended way to reach it in 2026 is the ML Kit GenAI APIs. The latest model, nano-v3, leads on the Pixel 10 series, with wider support across Pixel 8+, Galaxy S24+, and select Xiaomi and Motorola devices.

What you get out of the box

Ready-made GenAI tasks. ML Kit exposes Summarization, Proofreading, Rewriting, and Image Description as high-level calls tuned for quality on Nano. No prompt engineering, no model download, no MLOps. This covers the majority of “make this text shorter / cleaner / different tone” product features.

The Prompt API for custom work. When the canned tasks don’t fit, the Prompt API lets you send your own instructions to Nano on-device. It performs best on the Pixel 10 (nano-v3). Use it for classification, extraction, or grounded Q&A over local content — all without a network call.

On-device RAG. Pair Nano with on-device embeddings from the Google AI Edge SDK, retrieve chunks from a local knowledge base, and feed them in for grounded answers. The whole loop stays on the phone.

Device coverage and the fallback plan

There’s a catch worth planning around: the newest Gemini Nano 4 needs at least 12GB of RAM plus a supported flagship SoC, which as of mid-2026 narrows to devices like the Pixel 10 lineup, the Galaxy S26 series, and a handful of high-end Oppo, OnePlus, and Xiaomi phones. Older and mid-range devices either run an earlier Nano or none at all. Always check availability at runtime, and degrade cleanly: fall back to a bundled LiteRT-LM model (Gemma INT4) or to cloud Gemini when Nano isn’t present.

ONNX Runtime Mobile: when to pick it over LiteRT

Answer first: pick ONNX Runtime Mobile when your team is PyTorch-first or you need the exact same model binary on iOS and Android. ONNX is a framework-neutral model format; ONNX Runtime Mobile is a lean interpreter for Android, iOS, and embedded targets, with a QNN execution provider for Qualcomm NPUs.

LiteRT vs ONNX Runtime Mobile

LiteRT wins if: you’re already in the TensorFlow world, you want official Google support and MediaPipe integration, or you need the Gemini Nano path. Hexagon and GPU delegates are mature and first-party.

ONNX Runtime Mobile wins if: your model is PyTorch or ONNX-native, you want one binary across iOS and Android, or you’re pulling models from Hugging Face that export ONNX directly. Snapdragon NPU support via the QNN provider is solid.

Raw speed is a wash: both land around 30–50ms on MobileNet on flagship hardware, and both support INT8, INT4, and dynamic quantization. The real difference is ecosystem gravity — LiteRT has more Android-first docs, pretrained models, and the MediaPipe on-ramp.

NNAPI is deprecated: what to do instead

Answer first: don’t start anything new on NNAPI, and migrate legacy code to LiteRT. The Android Neural Networks API was the old bridge between frameworks and hardware accelerators. Google introduced it in Android 8.1 (2018) and deprecated it in Android 15.

Why it’s going away

NNAPI was a lowest-common-denominator interface, and vendors implemented it unevenly, which fragmented the story. Transformer and diffusion models also move faster than a framework API baked into each OS release can keep up with. Google’s answer is an updatable runtime that ships outside the OS: LiteRT (delivered as TensorFlow Lite in Google Play Services) for custom models, and AICore for Gemini Nano.

The migration paths

For TensorFlow models: move to LiteRT with the GPU delegate (and Hexagon where available). Swapping an NNAPI delegate for the LiteRT GPU delegate is close to a one-line change.

For ONNX models: use ONNX Runtime Mobile with the QNN execution provider on Snapdragon.

For GenAI: use AICore and the ML Kit GenAI APIs rather than trying to run a foundation model through the old accelerator API.

Model quantization: FP32 to INT4 tradeoffs

Quantization stores weights and activations in lower precision to shrink the model and speed up inference. The ladder runs FP32 → FP16 → INT8 → INT4, and the newest NPUs add INT2 and FP8 at the bottom. Each step trades a little accuracy for size and speed.

The practical tradeoff, level by level

FP32 (baseline). Full 32-bit precision. Best accuracy, biggest and slowest. This is a training format, not a shipping one.

FP16 (half precision). Roughly half the size, accuracy loss usually under 0.1% on vision. A safe compatibility step that runs on older GPUs.

INT8 (post-training quantization). About a 75% size cut versus FP32, ~35ms on CPU and ~15ms on an NPU for a MobileNet-class model, with 0.5–2% accuracy loss on ImageNet-scale tasks. This is the production default for vision, and it’s almost always acceptable.

INT4 (aggressive). Roughly an 87.5% size cut, accuracy loss 1–5% depending on the model and calibration data. Marginal for dense tasks like segmentation, but it’s what makes on-device LLMs fit at all — a Gemma-class model in INT4 lands in the low gigabytes instead of tens.

Which quantization method to use

Post-training quantization (PTQ): quantize a trained model without retraining. Fast, needs representative calibration data, great for INT8. This is what the LiteRT converter and MediaPipe use by default.

Quantization-aware training (QAT): train with simulated low precision so the model learns to tolerate it. Costs 10–50% more training time and buys back 1–2% INT4 accuracy. Worth it for accuracy-critical domains like medical or accessibility.

Dynamic quantization: quantize weights, keep activations in float. Smaller model with no calibration step; handy for RNNs and Transformers where per-layer precision matters.

Hardware acceleration tiers: what runs where in 2026

Not every Android phone is a flagship. The install base splits into four tiers, and your app has to behave on all of them. Plan for the floor and let the ceiling be a bonus.

Figure 3. Four device classes, four capability envelopes. The gap between Tier 1 and Tier 4 is the size of your fallback logic.

Tier 1: flagship NPU (Snapdragon, MediaTek)

Chips: Snapdragon 8 Elite Gen 5 (Hexagon NPU, ~37% faster than the prior generation, up to ~220 tokens/sec local decode, INT2 and FP8 support), MediaTek Dimensity 9500. Envelope: INT4 LLMs up to ~12B, dense vision in 20–40ms. This is where on-device chat is genuinely usable.

Tier 2: Tensor TPU (Google Pixel)

Chips: Tensor G5 (Pixel 10, Google’s first TSMC-built chip) and Tensor G3/G4 (Pixel 8/9). Envelope: similar vision latency to Tier 1, plus native AICore and Gemini Nano (nano-v3 on Pixel 10). If your feature is GenAI-first, this tier gives you the platform LLM for free.

Tier 3: mid-range GPU (Adreno, Mali)

Chips: Snapdragon 7 series, MediaTek Dimensity 6000/7000. No dedicated NPU; the GPU delegate is your best bet. Envelope: INT8 vision up to ~200MB at 40–70ms. On-device LLMs are impractical here — route them to the cloud.

Tier 4: budget / older (CPU-only)

Chips: Snapdragon 4 series, Android Go devices, older Cortex-A cores. CPU inference only. Envelope: tiny INT8 models under 50MB (keyword spotting, small detectors) at 100–300ms. The right move is graceful degradation, not pretending the NPU exists.

Comparison matrix: runtimes head-to-head

One correction worth flagging if you’re reading older guides: ExecuTorch is no longer experimental. It shipped a stable 1.0 and is on the 1.x line in 2026, with a Maven Android package and QNN/Vulkan backends. It’s a real option for PyTorch-native teams, not a research toy.

What can actually run on-device in 2026

Grounded in model sizes and latencies you can ship today on a flagship NPU.

Vision

Object detection (SSD MobileNet v2). 4MB INT8, 30–40ms, 90 COCO classes. Surveillance anomalies, barcode scanning, AR perception.

Face detection and landmarks. 5–10MB, 15–20ms, 468 keypoints. Proctoring, filters, landmark-based emotion cues.

Hand landmarks. 4–8MB, 10–15ms, 21 keypoints. Sign language, gesture control, hand-exercise tracking.

Pose estimation. 4–15MB, 25–40ms, 17–33 keypoints. Fitness, physical therapy, fall detection.

Segmentation. 10–30MB INT8, 50–100ms. Background blur, scene understanding, medical image work.

OCR. 20–50MB (detect + recognize), 100–300ms. Document scanning, receipts, accessibility.

Audio and speech

Speech recognition (Whisper-tiny). ~75MB, 1.5–2× realtime on a flagship, 99 languages. Captions, voice commands, transcription. Needs the model pre-downloaded.

Audio classification. 5–20MB, 10–50ms. Sound context for accessibility, smart-home triggers, monitoring.

Text-to-speech (small local voices). 40–100MB, ~0.5–2s per utterance. Lower naturalness than cloud TTS, but offline. Accessibility, audiobooks, read-aloud.

Text and LLM

Text embeddings (MobileBERT, DistilBERT). 20–50MB, 100–200ms per chunk. On-device semantic search, RAG, dedup.

Small LLMs (Gemma INT4). Low single-digit gigabytes; a few tokens/sec on a mid NPU and up to ~220 tokens/sec on a Snapdragon 8 Elite Gen 5. Chat, summarization, writing help. Context caching cuts repeat-query latency.

Gemini Nano (platform LLM). nano-v3 on Pixel 10, baked into the OS via AICore, called through ML Kit GenAI. Summarize, proofread, rewrite, and custom prompts — all offline on supported devices.

Privacy and compliance: why on-device is HIPAA/GDPR shorthand

Healthcare and fintech leads keep asking the same question: can we send this data to a cloud API? For biometric and health data the honest answer is “not without a lot of paperwork.” On-device inference sidesteps most of it because the sensitive bytes never leave the phone.

HIPAA

A teledermatology app that ships patient photos to a server for AI triage holds Protected Health Information the moment those photos arrive. That pulls in a Business Associate Agreement with your cloud provider, encryption in transit and at rest, breach notification, and audit logging. Run the same triage model on-device and only the result (“refer to a dermatologist”) leaves the phone. HIPAA still touches that metadata, but the PHI never hit your infrastructure, and your compliance surface shrinks accordingly.

GDPR and biometrics

GDPR Article 9 treats facial data as special-category, which demands explicit consent and a defensible lawful basis. Sending faces to a cloud emotion-detection API means a Data Processing Agreement and a legitimate-interest assessment across jurisdictions. Process face landmarks on-device and no biometric data flows off the phone at all — consent is still required, but enforcement is obvious because the app can’t phone home without network permission.

PCI-DSS

An app that verifies card details on-device keeps unencrypted card data out of your servers, which pulls large chunks of your systems out of PCI-DSS scope. Less data in transit means a smaller audit and a shorter compliance bill. If your team is weighing this, our AI integration engineers have shipped compliant on-device pipelines and can sanity-check the architecture before you commit.

Cost math: on-device vs cloud, worked through

Take a fitness app with 100K monthly active users, each running 30 inferences a day (movement detection, form checks, rep counting). That’s 3M inferences a day.

Cloud inference

Assume a conservative $0.002 per prediction for a lightweight hosted vision model (managed inference runs roughly $0.001–$0.02 depending on model and provider). The arithmetic:

• 3M inferences/day × $0.002 = $6,000/day
• $6,000/day × 30 = $180,000/month
• Annual inference: ~$2.16M
• Plus monitoring, egress, and compliance tooling on top

On-device inference

• Training and quantization: a one-time engineering cost.
• Model delivery: an 8MB model × 100K users = 800GB/month; at ~$0.12/GB that’s about $96/month.
• Telemetry server: ~$500/month for logging and feedback.
• Inference: $0 marginal — it runs on the user’s device.
• Recurring total: under ~$600/month, roughly $7K/year plus the one-time build.

Break-even

Cloud runs into the millions per year at this volume; on-device recurring cost is a few thousand. The real price of on-device isn’t compute — it’s the engineering: quantization, testing across 30–50 device types, and fallback logic for the low tiers. Above roughly 100K users at 10+ inferences each per day, on-device is the economically obvious choice. Because we build with Agent Engineering, that one-time engineering cost lands lower and faster than a typical hand-built pipeline, which pulls the break-even point in further.

Mini case: on-device anomaly detection in NetCam

NetCam is a video surveillance platform we built for SMB security — parking lots, warehouses, retail. The brief: flag unusual activity (abandoned bags, loitering, tailgating) in real time without streaming every frame to the cloud.

Before: cloud-only

Every frame went to a cloud model at 10 fps. At roughly 2MB per frame that’s ~1.7TB per camera per day, which at $0.12/GB works out to about $74K/year per camera in bandwidth alone — before compute or storage. Alerts landed 200–300ms after the event (network plus inference), too slow to trigger a gate or reach a guard in time.

After: on-device + cloud hybrid

We quantized SSD MobileNet to INT8 (4MB) and ran it on the Android device driving the camera. Detection happens locally at 30–40ms per frame; only confirmed anomalies push a low-res thumbnail plus metadata to the cloud.

• Bandwidth: ~$262/year per camera, down from ~$74K — a roughly 280× cut.
• Latency: ~140ms end to end (40ms local + ~100ms upload), so alerts arrive inside ~200ms.
• Reliability: the camera keeps detecting when the network drops, storing anomalies to sync later.
• Privacy: raw video never leaves the camera; only detections sync, which satisfies on-premise data-residency rules.

Figure 4. NetCam per-camera economics, cloud-only vs on-device hybrid. Same detector, moved to the edge.

The business outcome: NetCam scales to 50+ cameras per site without bandwidth blowing up, and enterprise buyers (logistics, airports) adopted it because sub-200ms alerts are fast enough for automated response. A cloud-only design couldn’t hit that latency or that price. Want a similar assessment of your own video stack? Book a 30-minute call and we’ll walk the numbers. There’s more on the detection side in our AI video surveillance guide and the deeper AI security camera engineering playbook.

Decision framework: five questions to pick on-device vs cloud

Run any new feature through these five. The more you answer “yes,” the more on-device wins.

1. Is it latency-critical (under 200ms)? If yes, on-device wins — the network alone eats 50–500ms. Detection, landmarks, and pose all qualify.

2. Is the data sensitive (biometric, health, financial)? If yes, on-device is close to non-negotiable, and it makes HIPAA and GDPR cheaper to satisfy.

3. Must it work offline? If yes, on-device is the only option that survives airplane mode and dead zones.

4. Does the model fit (under ~500MB vision, ~2GB LLM)? If yes, on-device is feasible. If not, split it: edge for the latency-critical part, cloud for the rest.

5. Do you control the release cadence? If it’s your app, on-device model swaps are easy (download a new file at startup). If a platform owns the model, lean on Gemini Nano instead.

Three or more “yes” answers point to on-device. Five out of five and it’s mandatory. All “no” and plain cloud inference is the simpler call — take it.

Five pitfalls in production Android ML

1. Accuracy craters on real-world data. Your INT8 model scores 95% on the validation set and 72% on customer photos shot in bad light. The fix: collect 500–1,000 real samples from the actual deployment, measure against them during development, and retrain with augmentation (blur, noise, brightness) if it’s not good enough.

2. Out-of-memory crashes on cheap devices. A 150MB model is fine on a flagship and dies with an OutOfMemoryError on a 3GB phone. Profile on the low tier (a Galaxy A-series, a Redmi Note), and shrink via INT4, pruning, or distillation. Rule of thumb: keep the model under ~30% of target RAM.

3. Silent CPU fallback. You target the NPU, but on an older chip the delegate quietly fails and drops to CPU. Latency jumps from 30ms to 200ms and the UI stutters. Log every delegate decision, test on real target hardware (not just the emulator), and add explicit fallback with timeouts and warnings.

4. No model update strategy. You bundle the model in the APK, then improve it — and shipping the new one now means a full store release and a 1–2 week rollout while old versions run stale. Store models in cache or cloud storage, update silently in the background, version them, and track the active version in prefs.

5. Battery drain from always-on inference. Pose detection every 100ms in a background service can pull 15–20% battery in an hour, and users notice. Batch inferences, gate them behind an accelerometer motion check, and throttle in low-power mode.

KPIs: what to measure once you ship

Quality KPIs. Track field accuracy against ground truth on a 500-sample set every two weeks, and watch false-positive and false-negative rates separately — the trade between them is a product decision. Set a floor (say, 90% top-1) that triggers a retrain or rollback.

Business KPIs. Watch inference volume, cache hit rate, and latency at p50/p95/p99. A p99 creeping past 200ms usually means delegate failures or device overload, not a modelling problem.

Reliability KPIs. Track OOM crash rate, ANR rate (inference blocking the main thread), and the share of inferences that actually reached the NPU versus falling back to CPU. Wire these to Crashlytics or Sentry and set targets: OOM under 0.1%, ANR under 0.5%, CPU fallback under 20%.

When NOT to ship on-device AI

The model is bigger than the phone. A 13B+ model in higher precision won’t fit in storage or RAM on most devices. Unless you’re targeting a narrow band of 12GB+ flagships, that’s a cloud job.

You retrain constantly. If accuracy drifts daily and you need fresh weights every 24 hours, pushing model updates to every device is a losing race. Cloud inference with a daily refresh is cleaner.

The feature isn’t time-sensitive. A weekly “your top focus areas” digest has no user-visible latency, so there’s no reason to fight device fragmentation for it. Compute it server-side.

You can’t afford device testing. On-device ML means validating across 30–50 device and OS combinations. With a two-person team and a tight deadline, cloud carries less risk. If you want that testing handled, that’s the kind of thing our custom software development team does as a matter of routine.

FAQ

What is on-device AI on Android?

On-device AI runs a neural network directly on the phone’s CPU, GPU, or NPU instead of a remote server. On Android in 2026 the main stack is LiteRT 2.x for the runtime, MediaPipe Tasks for vision and audio, and Gemini Nano via ML Kit for language. The payoff is privacy (data stays on the device), sub-100ms latency, and offline operation.

What is LiteRT, and is it different from TensorFlow Lite?

LiteRT is Google’s renamed and extended TensorFlow Lite (renamed September 2024). Your .tflite models still load unchanged. The main addition in LiteRT 2.x is the CompiledModel API for hardware acceleration, plus LiteRT-LM for running on-device language models.

Is NNAPI dead? Should I migrate?

NNAPI is deprecated as of Android 15 but still works for backward compatibility. Existing Android 10–14 code keeps running. For anything new, use LiteRT delegates (GPU, Hexagon); moving from an NNAPI delegate to the LiteRT GPU delegate is close to a one-line change.

Which devices support Gemini Nano?

Gemini Nano runs on-device across Pixel 8+, Galaxy S24+, and select Xiaomi and Motorola phones, with nano-v3 leading on the Pixel 10 series. The newest Gemini Nano 4 needs at least 12GB of RAM plus a flagship SoC, which as of mid-2026 means devices like the Pixel 10, Galaxy S26, and a few high-end Oppo, OnePlus, and Xiaomi models. Always check availability at runtime and provide a fallback.

How large a model can run on Android?

Vision models run 4–200MB in INT8. Small LLMs like Gemma in INT4 land in the low single-digit gigabytes; LiteRT-LM handles Gemma-class models up to about 12B on top-tier NPUs. Flagships have 8–16GB RAM, budget phones 3–4GB. Keep the model under roughly 30% of target RAM and test on the device tier you actually support.

Is on-device AI free after the download?

Nearly. There are no per-call cloud charges — only device energy (a few milliwatts per inference) and memory. The one-time costs are the model download (roughly $0.01–$0.10 of bandwidth per user) and modest telemetry servers. The main investment is engineering, not runtime cost.

How do I run Whisper speech recognition on Android?

Convert a Whisper model (the C++ port is a common starting point) to LiteRT or ONNX, quantize to INT8 (~75MB for the tiny model), and run it through the LiteRT interpreter or a MediaPipe audio pipeline. Expect 1.5–2× realtime on a flagship. The model must be pre-downloaded, so live transcription on budget phones is a stretch.

Is on-device AI different on iOS?

iOS has Core ML as its runtime and Apple Intelligence for platform GenAI, but the APIs differ from Android’s. For one codebase across both, ONNX Runtime Mobile or LiteRT (with iOS builds) give you the closest parity. Raw performance is comparable; the tooling and platform-LLM stories are not.

What to read next

On-device AI is the new default on Android

In 2022, on-device inference was a flagship luxury. In 2026 it’s the default for anything touching sensitive data, needing sub-100ms latency, or running offline. LiteRT 2.x and MediaPipe Tasks have matured to the point where not using them is the exception, and NPUs from Snapdragon, Tensor, and MediaTek have made INT8 inference several times faster than CPU-only.

The real cost is engineering: quantization, testing across the device tiers, fallback logic, and KPI monitoring after launch. The payoff is a big cost cut at scale, predictable low latency, and compliance that’s far easier to defend.

We’ve shipped on-device AI into production Android apps across surveillance, proctoring, and edge robotics. We know which tools scale, which don’t, and where the gotchas hide. If you’re scoping an on-device feature, we’ve already solved the hard parts.