LLM · Edge AI · Efficient Inference
Offline African-Language LLM
Built an offline LLM system designed to operate on commodity hardware while supporting African-language interaction under strict compute and memory constraints.
System flow
Metrics
- Model
- Qwen2.5-1.5B-Instruct
- RAM
- ≤ 8 GB
- Tokens/sec
- 8–12 tok/s (CPU)
- Model Size
- ~1.1 GB (Q4_K_M)
- Accuracy
- In evaluation
Research angle
How do model size, quantization, grammar constraints and hardware limitations affect accuracy and inference efficiency?
Problem
Most consumer AI assistants assume constant connectivity, cloud-hosted GPUs, and English-first language coverage — none of which hold for a large share of everyday computing environments, where connectivity is intermittent, hardware is commodity-grade, and users move fluidly between English and local languages.
Why it matters
If useful AI systems only run in the cloud, access becomes a function of bandwidth and hardware budget rather than need. An offline-first system decouples usefulness from infrastructure — it can run on a shared family laptop, in a low-connectivity region, or in a classroom with no reliable internet, while still respecting the languages people actually speak.
Constraints
- CPU-only inference — no dedicated GPU assumed
- ~8GB RAM ceiling, shared with the OS and other applications
- Must stay responsive enough for real conversational use on commodity laptops
- Small quantized models must still emit syntactically valid structured output for tool calls
- Limited training-data coverage for African-language prompts and code-switching
Approach
The system pairs a lightweight agent orchestrator with a quantized LLM served through llama.cpp in GGUF format. Rather than trusting the model to emit well-formed tool calls on its own, generation is constrained at decode time with a formal grammar (GBNF), forcing every output into a valid schema before it reaches the tool layer. This shifts reliability from 'does the model behave' to 'can the model produce anything invalid' — the latter becomes structurally impossible.
Experiments
- Comparing quantization levels (Q4_K_M vs Q5_K_M vs Q8_0) across latency, RAM footprint and output quality
- Grammar-constrained decoding vs. free-form generation, measured on tool-call validity rate
- Throughput and memory profiling across model sizes (0.5B / 1.5B / 3B parameters) on an 8GB machine
- Prompt scaffolding strategies for improving African-language response quality without fine-tuning
Results
- Grammar constraints reduce malformed tool-call output to effectively zero, independent of model size
- Q4_K_M offers the best latency/RAM/quality tradeoff for the 1.5B model class on 8GB hardware
- RAM headroom — not raw CPU throughput — is usually the first constraint hit on commodity laptops
Lessons
- Small quantized models are usable for orchestration and tool-use even when raw generation quality is mediocre — structure matters more than fluency for agentic tasks
- Grammar constraints add measurable decode-time overhead, so grammar complexity itself becomes a tuning parameter
- Designing for 8GB RAM forces architectural discipline that pays off even on higher-spec hardware