Over the past months, we have deployed small language models — most under 500M parameters, all under 2B — on Raspberry Pi hardware for two production use cases: a legal assistant for conflict-related sexual violence (CRSV) survivor networks, built with Bibliothèques Sans Frontières and the Dr. Denis Mukwege Foundation, and grounded question-answering over a French primary-school curriculum from Senegal. Both are knowledge-work applications — legal analysis, lesson preparation — of the kind usually assumed to require cloud-hosted models, running in environments where cloud infrastructure is not an option.

This post describes the two engineering results that made these deployments work. First, small models trained as reasoning models — with explicit reasoning traces and native citations — provide the auditability that high-stakes offline use requires, at a fraction of the size usually assumed necessary. Second, Cache-Augmented Generation (CAG) makes long contexts practical on edge hardware by turning them into persistent, precomputed objects: in our benchmarks, prefill time on a 21,000-token corpus drops from 19 minutes to 15 seconds.

We conclude with what these results suggest about the direction of small-model development: not one general-purpose small model, but many specialized ones.

Knowledge work where the cloud is unavailable

Both deployments run fully offline by design, for two distinct reasons.

For the CRSV assistant, the primary constraint is confidentiality. The system serves survivor networks and their advocates in conflict-affected regions, and queries in this context can be testimony-adjacent — they may reveal who is asking, about what, and from where. Local inference is the security model: sensitive data never leaves the device. Connectivity is the second constraint, since many regions where the Red Line Initiative's Guidebook is most needed lack reliable internet infrastructure.

For the education deployment, the constraints are cost and connectivity: rural schools with intermittent network access, and hardware budgets where a Raspberry Pi is the realistic ceiling.

It is worth underlining that neither task is simplified to fit the hardware. Navigating an international legal framework and synthesizing across a curriculum are knowledge work in the full sense — the kind of workload the industry currently routes to hosted frontier models by default. The constraint here is infrastructure, not task complexity. Whatever capability the system has must run on a CPU, within a few hundred megabytes, and it must be complete: there is no fallback endpoint when the model is unsure.

A reasoning model on device

The model at the center of the Redline deployment is a 321M-parameter specialist based on the architecture of Baguettotron, our reasoning-focused small model trained with the SYNTH pipeline. Rather than a scaled-down chatbot, it produces an explicit analytical trace before answering — and runs the full reasoning-and-quoting pipeline at 28 tokens per second on a Raspberry Pi 5.

Given a query and retrieved sources, the model first generates a source analysis: it decomposes the query, assesses what each source establishes, and identifies the applicable legal framework. It then drafts an answer in which every substantive claim carries an inline citation quoting the exact passage of the Guidebook it relies on.

In this deployment, reasoning traces and citations are not a presentation feature but the safety mechanism. The users are advocates and legal experts operating offline, often without access to a second opinion, in situations where the stakes can be legal or evidentiary. Because the reasoning trace and the citations are exposed, a user can verify which section of the Guidebook an answer came from, follow the model's reasoning, and catch possible errors before acting on them. This is what makes a small model deployable in a high-stakes setting.

Two factors make 321M parameters sufficient here. The first is training methodology: synthetic training pipelines designed around grounding and citation behavior produce models that are unusually reliable at those tasks for their size. The second is data preparation. Most of the project's effort went not into the model but into converting the Red Line Guidebook — a legally dense, structurally complex document — into a corpus a small model can use reliably:

• Semantic chunking that follows the Guidebook's logical units rather than fixed token windows
• Context injection, annotating chunks with its jurisdiction and legal framework so the model does not conflate, for example, ICC jurisprudence with regional instruments
• Entity normalization across the document's many acronyms and legal terms
• Markdown structuring so document hierarchy survives retrieval

At this scale, data curation does much of the work that parameter count usually absorbs. The production system is deliberately simple: a Flask API over a LanceDB vector store and a GGUF model served through llama.cpp, packaged in Docker, available in English and French, streaming the source analysis and cited answer as they are generated.

Handling long context on device

The education deployment posed a different problem. The corpus includes questions that span multiple documents: comparisons, aggregations, syntheses. Answering them requires a whole thematic cluster in context, about 21,000 tokens for the largest cluster we tested.

On a Raspberry Pi, this is prohibitive with standard inference. Prefill — the forward pass that processes the context before generation begins — is compute-bound, and on the Pi's CPU it takes 1,154 seconds (about 19 minutes) to ingest 21,000 tokens with Qwen2.5-1.5B-Instruct. In a standard RAG setup, this cost is paid again for every question, since the context is re-sent and re-processed each time.

The key observation is that the grounding corpus is fixed: a curriculum or a legal guidebook does not change between questions. Cache-Augmented Generation exploits this by keeping the context as a stable prefix and persisting its key-value (KV) cache — the intermediate attention states computed during prefill. The cache is computed once; every subsequent question reuses it. llama.cpp supports this natively through prefix caching, and the server's cache counters let us verify that the full context is reused on each call.

We benchmarked this on the Pi across context sizes and two model scales:

The saving grows with context length, which is exactly the regime where edge hardware is weakest. In effect, CAG changes what a long context is: not a per-query processing cost, but a resident object — a few hundred megabytes of precomputed states that live on the device and can be queried at will. A natural extension, which we have not yet benchmarked, is to compute the cache on a fast machine and distribute the KV states to devices alongside the model weights, eliminating even the one-time prefill cost. A grounding corpus then becomes a shippable artifact, like a model checkpoint.

What CAG does not solve

Two limits remain, and they are worth stating plainly.

Decode is not cached. Every generated token attends over the full resident context, so generation speed falls from 15–20 tokens per second on short contexts to about 1.2 tokens per second (1.5B) over the 21k-token cache. Single-document use stays interactive; corpus-scale questioning is batch work on a Pi.

Model capability is the ceiling. At corpus scale, the 0.5B model degrades badly — repetition loops, hallucinated place names, and in one case denying it had been given the corpus at all. The 1.5B model behaves much better: no hallucination, and five of eight cross-document answers usable. But it still fails cross-document counting outright, reporting two documents on a topic when the correct answer is six. CAG makes the mechanics of long context nearly free; it does not make the model more capable.

When retrieval alone is enough

Not every knowledge-work task on the edge needs generation at all. For teacher lesson materials, the most useful output is often not a generated answer but the lesson sheets themselves — documents already written and validated by educators. For this workload we built a pure retrieval system: given a teacher's question in French, it returns the most relevant existing fiches, ranked. No model writes or rewrites anything, which also means the system cannot hallucinate by construction — the output is always an authored document.

The engineering question was the same one that runs through this post: how much model does the task actually need? Our default embedding model, BGE-M3, is excellent — multilingual, with dense, lexical, and late-interaction retrieval heads in a single model — but it weighs 2.2 GB and takes about 2.5 seconds per query end-to-end on a Raspberry Pi. So we ran a controlled comparison against multilingual-e5-small, a 449 MB dense encoder, holding everything else in the pipeline identical: the same corpus, the same 129 real teacher questions, the same fusion and reranking stages, the same LLM judge scoring relevance on a 1–5 scale.

The pipeline is where the retrieval quality lives, and it is worth describing. Each lesson sheet is embedded across five aspect fields — the pedagogical question it answers, its subject, title, learning objectives, and targeted competencies — each stored as a separate vector column. A query is matched against all five independently, the ranked lists are fused with weighted score fusion, and a compact ColBERT reranker (per-token late-interaction scoring) reorders the top candidates and drops weak matches.

With that pipeline held constant, the two encoders reach parity. BGE-M3 is marginally better on the single top result (mean relevance 3.70 vs. 3.62 out of 5); e5-small is better across the top two results (3.65 vs. 3.52) and misses far less often — both top results are weak on only 2.4% of questions, against 5.6% for BGE-M3. Neither difference exceeds 0.13 on a 5-point scale. On the Pi, the swap cuts query embedding from 850 ms to 68 ms, roughly 12× faster, at one-fifth the size.

The practical consequence is reach. Quantized to INT8, e5-small fits in about 33 MB — roughly 70× smaller than BGE-M3 at full precision — which allowed us to ship the same retrieval logic in two offline builds: the full five-field pipeline with reranker on the Raspberry Pi, and a memory-trimmed Android build (three aspect fields, no reranker) for older phones. The trimmed build gives up measurable quality (mean top-2 relevance drops from 3.65 to 3.19), but over half of its top results remain highly relevant — usable retrieval, running entirely offline, on hardware that fits in a teacher's hand.

The lesson mirrors the Redline result one layer down the stack: invest in the pipeline — multi-aspect embeddings, fusion, domain-tuned reranking — and the individual model can shrink dramatically without the system losing quality.

Toward many small models, not one

Taken together, these results point to a tension. The training recipe that produces auditable, citation-grounded reasoning favors one prompt ordering and one data design. Efficient long-context residency favors another ordering and a different architecture — fewer KV heads, moderate depth. And the two deployments described here need different training data entirely, before either constraint applies.

Within a sub-2B parameter budget, these requirements cannot all be satisfied by a single model. Larger models resolve such tensions with scale; small models resolve them with specialization. In our experience, smallness works when it is specific — a model built for a particular corpus, task profile, and deployment constraint.

Specialization also has a linguistic and cultural dimension that generalist models cannot deliver. Generalist training distributions are dominated by high-resource, predominantly English web data; languages, curricula, and legal traditions outside that distribution are represented thinly and generically, when at all. The deployments described here already sit outside it: French-language legal reasoning over an international framework, and a curriculum written for Senegal's education system, with its own geography, references, and pedagogical conventions. A generalist model can be approximately adequate for all of these contexts; it cannot be attuned to any of them. A small model trained on purpose-built data can — down to the terminology of a specific legal instrument or the structure of a national curriculum, and extending, in future work, to local languages that frontier training sets barely cover.

The economics support this direction. At frontier scale, a training run is a major capital expense. At 300M parameters, it is a pipeline that can be run repeatedly: per specialised agentic process, per national curriculum, per language pair. The same synthetic training infrastructure that produced Baguettotron produced the Redline specialist, and can produce the next specialist from a different guidebook, curriculum, or language at marginal cost.

We think this is the more promising trajectory for edge AI: training a diverse population of small, specialized, auditable models — multilingual and culturally attuned where their users are, owned by or built for the institutions that need them, with their grounding corpora distributed alongside them as precomputed caches. The remaining barriers for school systems, health networks, and survivor organizations to operate models of their own are organizational rather than technical. The technical part now fits on a Raspberry Pi.

---

• A ready-to-use API of our SLM for Raspberry can be found here:  https://huggingface.co/PleIAs/Pleias-SLM-RAG
• The extended Redline repository with the training dataset and other models: https://huggingface.co/datasets/PleIAs/BSF_Redline
• A Raspberry ready librarian CAG app with Qwen2.5 1.5b built in: https://github.com/Pleias/pi-cache-augmented-generation/pkgs/container/cag-app