On-Device AI · React Native · LLM Systems
Zayla
An offline-first mobile AI assistant that runs LLM inference locally on constrained Android hardware — zero servers, zero accounts, zero data collection.
System flow
App screenshot
Metrics
- Model
- User-selected GGUF (device-scored)
- Quantization
- Q4_K_M / Q5_K_M
- RAM usage
- Model-dependent — scored pre-download
- Performance
- Benchmarking in progress (live in-app llama-bench)
- Data collection
- Zero — fully offline
Engineering challenge
Running useful LLM inference locally while staying inside mobile memory, thermal and compute constraints — without a server to fall back on.
Problem
On-device chat apps that claim to be 'privacy-first' often still phone home for account creation, crash analytics, or model hosting. Genuinely offline LLM inference on commodity Android hardware — with real model management and real acceleration — is a much harder systems problem than wrapping a cloud API.
Why it matters
Local inference means privacy by construction (nothing to intercept because nothing is sent), offline operation in low-connectivity conditions, no server costs or uptime dependency, and predictable behavior on hardware the user already owns rather than whatever a cloud provider happens to run.
Constraints
- No server component at all — chat, model download, translation and benchmarking all have to work fully offline once a model is on-device
- Wide hardware variance across Android devices — RAM, storage and GPU support differ enormously, so the app has to recommend the right model per device rather than assume a spec
- Native bridge complexity — llama.cpp is C++; React Native needs a stable, crash-resistant bridge (llama.rn) to drive it from JS
- Model files are multi-gigabyte downloads over unreliable mobile networks — downloads must resume, retry, and verify integrity
Approach
Built in React Native/TypeScript with a native bridge (llama.rn) into llama.cpp, so no request ever reaches a server. A model-management layer scores GGUF models against the device's actual RAM and storage before recommending a download, then handles that download as a resumable, checksum-verified queue. On supported GPUs, inference can offload from CPU for a speedup over CPU-only decoding, and a bundled llama.cpp benchmark harness (llama-bench) lets a user measure real prompt-processing and token-generation throughput on their own hardware instead of trusting a fixed number that wouldn't apply to their device anyway.
Experiments
- Comparing CPU-only vs. GPU-offload inference paths across supported chipsets
- Tuning quantization level (Q4_K_M vs Q5_K_M) for the throughput/quality/RAM tradeoff on 8GB-class devices
- Stress-testing the model-download queue for concurrency bugs under retries and app backgrounding
Results
- Fully functioning offline chat pipeline — zero network calls once a model is downloaded
- Resumable, checksum-verified downloads eliminated corrupted-model failures from interrupted transfers
- Live in-app benchmarking (llama-bench) replaces a single fixed performance claim with a real, device-specific measurement
Lessons
- Real privacy claims have to be architectural, not just policy — 'we don't log your chats' is a much weaker guarantee than 'there is no server for your chats to reach'
- Device-aware recommendations matter more than raw model quality — the best model is the one that actually runs on the user's phone without crashing
- A device-specific benchmark beats a single headline number — performance on this class of hardware varies too much for one figure to be honest