The NPU didn’t make it faster

Croplock runs its language model on the box in your house. No cloud, no account, no API key. That constraint is the product, but it also means the hardware question is not which GPU — it is what a fanless, sealed, always-on single-board computer can actually sustain. This is what we measured, including the part that came out against our expectations.

The board is an Orange Pi 5B: Rockchip RK3588S, 16 GB, four Cortex-A76 cores at 2.4 GHz alongside four A55s at 1.8 GHz, and a three-core NPU. The model is Qwen2.5-3B-Instruct at W8A8, served through RKLLM 1.2.3 on RKNPU driver 0.9.8. The CPU baseline is the same model at q4_k_m under llamafile and under Ollama on the same board.

The result we did not expect

Eight runs — three with the board cooled below 58 °C first, then five back-to-back with no cooldown at all.

The NPU is tied with Ollama and about 12% behind llamafile’s cold start. Whatever else the NPU is doing, it is not generating tokens faster. If you are choosing a single-board computer on the strength of a TOPS figure and you intend to run a chat workload on it, that number is not going to buy you what you think it will.

Prefill is the opposite story

Prompt processing on the NPU ran at 85 to 128 tok/s, and it climbed as the cache filled. The parallelism is real; it just lives in prefill rather than in single-token decode, which is inherently serial.

That distinction turned out to decide the whole question, because our workload is not chat. Every answer the assistant gives is grounded on facts retrieved from the device’s own records, so the prompt is long and the reply is short. When we cut over from the CPU path in July, grounded questions went from two to three minutes down to seven to fifteen seconds. Decode was a wash, so prefill is where that came from — though we should be precise about what we know: we measured the end-to-end latency and the NPU’s prefill rate. We did not separately measure the CPU’s prefill rate, so the mechanism is an inference from those two facts rather than a third measurement.

The general shape is worth stating plainly. Which half of the workload dominates decides whether an NPU is worth anything to you. Long prompt, short answer: large win. Short prompt, long answer: none.

Why we kept it anyway

Two reasons, and neither one is speed.

Thermal. Across five back-to-back runs the NPU held 6.18 to 6.23 tok/s — under 1% spread — while the SoC plateaued around 63 °C. The CPU runtimes reached 83–85 °C and lost throughput to heat soak. Run uncontrolled back-to-back, llamafile decayed from 8.0 to 5.1 tok/s and Ollama from 5.0 to 4.0. On a desk with a fan, cold-start numbers are what you feel. In a sealed passive enclosure that never turns off, sustained throughput is the only number that exists — and there the NPU wins outright.

Core availability. RKLLM runs on the little A55 cores. All four A76s stay free for the application — which on this box is a web server, a database, a vision cascade and an MQTT broker. A benchmark that measures inference alone will never show you this, and on a machine doing several things at once it is worth more than the tok/s.

A published benchmark that did not reproduce

We went into this partly because of a 2026 single-board-computer inference paper reporting that llamafile delivered four times the throughput of Ollama at 30–40% lower power on a Pi 5, at the same quantization. That is a large enough claim to act on.

It did not hold here. On RK3588 llamafile’s edge over Ollama was about 14%, not 4×, at 37% more RAM, and it was thermally bound. We are not disputing the paper on its own hardware; we are saying the result did not transfer. Different system-on-chip, different cooling, different measurement window. A cold-start benchmark on an actively cooled board tells you very little about a passively cooled one.

Our own power figure, in the interest of not making the same mistake: we do not have one. There is no inline meter on this bench, so the energy advantage that a twenty-degree thermal gap implies is unmeasured, and we are not going to assert it.

The failure that cost eleven hours

The assistant went down and stayed down for eleven hours before anyone noticed, because two separate health checks both reported it healthy.

The first theory was memory exhaustion: the vision cascade keeps fifteen RKNN models resident, and fifteen sounds like a lot. It was wrong. They are three to four megabytes each — about 56 MB in total. A count is not a cost, and that instinct sent us the wrong way first.

The real mechanism was fragmentation, not exhaustion: roughly forty-five small, long-lived cascade buffers against the vision model’s single 2.94 GB contiguous request. The tell is in the error. The allocation fails with errno 14 EFAULT, not ENOMEM — an address-space problem wearing the costume of a memory problem. If you are staring at free memory you will not find it. We reproduced it deliberately by driving one vision cycle and watching the assistant’s allocation go 3062.8 MB, then zero, then 3062.8 MB again.

The instrument for this is /sys/kernel/debug/dma_buf/bufinfo, counting only the drm exporters. We had looked in rknpu/ and concluded no instrument existed. It was one directory further along.

As for the eleven hours of silence: one health check compared a model-alias string, and the other short-circuited on the warm-up prompt before the request ever reached the model. Both returned healthy against a dead engine. The probe now sends a prompt chosen specifically so that it cannot take that short circuit. A health check that can pass without touching the thing it checks is not a health check.

Quantization moves behaviour, not just quality

One more that cost us a surprise. Moving from q4_k_m to W8A8 preserved the model’s citation discipline — it still anchored its answers on real retrieved values — but it lost a self-refusal behaviour the earlier build had, where it would decline a hypothetical it had no data for. Nothing in a perplexity score would have caught that. Every model, quantization or runtime change now goes through a fixed set of behavioural cases before it ships.

What we would tell someone starting this

• Benchmark prefill and decode separately. They answer different questions, and an NPU moves them in opposite directions.
• Benchmark sustained, not cold. If the device is sealed, the first run is the least representative number you will ever take.
• Count what the inference leaves free, not only what it consumes. Which cores it runs on is a real number.
• Check the driver version early. Ours had to be brought up from RKNPU 0.9.3 to 0.9.8 before the NPU path would run at all.
• Treat a quantization change as a behaviour change until a regression suite says otherwise.

Sources

Throughput, thermal and RAM figures are from our own bench of 2026-05-31 on an Orange Pi 5B (RK3588S, 16 GB), RKLLM 1.2.3, W8A8 Qwen2.5-3B, RKNPU driver 0.9.8, measured from the runtime’s own RKLLMPerfStat counters and thermal_zone0 — three cooldown-controlled repetitions plus five back-to-back, 128 maximum new tokens, 512 context. The CPU baseline is from the same board on 2026-05-29, Qwen2.5-3B at q4_k_m under llamafile and Ollama. The four-times llamafile claim is from “LLMs Inference on SBCs” (arXiv:2511.07425) and is cited, not reproduced. The outage post-mortem is our own, from 2026-09-04. Power is unmeasured throughout.