Africa LLM (AFRI-LLM-CODEX)

Problem

Developers in low-bandwidth or offline environments across Africa need fast, private, on-device code-assistance tools that run reliably on budget hardware (2–4 CPU cores, 8GB RAM, no GPU) — with no guarantee of a stable power or internet connection.

Why it matters

Most coding assistants assume a cloud API and a reliable connection. If useful code assistance only works online, it's unavailable exactly where compute and connectivity are scarcest. Built for the Africa Deep Tech Challenge 2026's Laptop LLM Challenge, this project treats those constraints as the actual design brief, not an afterthought.

Constraints

• Hard ceiling of 8GB RAM and no GPU — the reference judging laptop is a 2-core/4-thread CPU with no dedicated graphics
• Zero network calls permitted at runtime — model, context extraction and translation all have to run fully offline
• The competition's scoring formula weights accuracy at 50%, throughput at 30% and efficiency at 20%, with a thermal penalty, so raw speed isn't the only axis that matters
• A model's declared parameter count is fraud-checked against the actual GGUF tensor count (±15% tolerance), so even metadata claims have to be accurate

Approach

The assistant runs Qwen2.5-Coder-1.5B-Instruct (GGUF, Q5_K_M) through llama.cpp with pinned thread affinity and a capped context window. Rather than dumping whole files into the prompt, a from-scratch dependency-graph engine (DS-Code Graph) parses the target codebase with Python's ast module (plus a regex-based scanner for C/H files), builds IMPORT/CONTAIN/CALL edges, and retrieves only the 1–2 hop neighbourhood around the function being patched — full body for the target symbol, signatures only for its dependencies. Output is constrained by a GBNF grammar so the model can only emit a valid {file_path, action, code_patch} JSON object, which also cuts generated tokens by roughly 80%. A separate NLLB-200 model handles translation into 8 African languages for a pedagogical tutor mode; it is never loaded at the same time as the coding model, so peak memory stays bounded by whichever model is larger, not their sum.

Experiments

• Benchmarked three quantization levels head-to-head (Q5_K_M, Q4_K_M, IQ4_XS) under identical cold-start conditions — Q4_K_M won on every measured axis, but IQ4_XS was rejected for being slower than Q5_K_M despite being the smallest file, since its format lacks optimized CPU dequantization kernels
• Reverted the initial Q4_K_M adoption back to Q5_K_M as a precautionary, accuracy-first choice once profiler data confirmed accuracy carries half the competition's scoring weight — the Q4_K_M vs Q5_K_M accuracy A/B test was never run, so the safer default was kept rather than assuming the speed gain was free
• Caught and corrected a parameter-count claim during fraud-check validation: the GGUF file's actual tensor count is 1.78B parameters, which fails a strict ±15% tolerance check against the model's own '1.5B' name
• Benchmarked NLLB-200 translation quality (chrF++) across all 8 supported languages to pick real coverage over assumed coverage

Results

• 0.68 accuracy on arc_easy (n=50, lm-eval-harness)
• 10.71 tokens/sec generation on CPU
• ~1.26GB peak RAM — well under the competition's 2GB hard ceiling and 8GB budget
• 100% offline runtime: zero network calls for parsing, context construction, or inference
• Translation quality benchmarked per language (chrF++): Swahili 58.0, Arabic 51.4, Zulu 51.0, Hausa 49.0, Igbo 40.0, Amharic 37.2, Yoruba 22.9, Tigrinya 22.6

Lessons

• Smaller isn't always faster — IQ-series quantization trades extra CPU compute for a smaller file, and without optimized kernels for that format, the compute cost outweighs the memory savings
• Grammar constraints are a latency lever, not just a correctness one — forcing structured output cut generated tokens by ~80%, which matters more on CPU than on GPU
• A dependency graph built in an afternoon (ast + regex, no embeddings, no vector DB) beats naive whole-file context for token-constrained, offline settings