Trained on close-ups, served whole plants

Croplock has a camera in the tent and a neural processor in the box, and the obvious thing to do with those is diagnose the plant. We built that. This is the record of what it took, what it got right, and the several ways it was confidently wrong — because the wrong turns are where the reusable engineering is, and because the honest state of the thing is that it is a good filter and not yet a good diagnostician.

The pipeline runs entirely on the RK3588’s NPU. A small image classifier — seventeen classes covering healthy, nutrient deficiencies by element, pests, disease, light burn and a few leaf-shape stresses — sits behind a chain of cheaper gates: is there a plant in frame, is it the species we trained on, is it a seedling. Each stage is a few megabytes at fp16 and the classifier scores an image in about eleven milliseconds. Nothing leaves the LAN. A language model narrates the result afterwards; it does not make the call.

Classification, not detection, and why

The first decision was to classify whole frames rather than draw boxes around lesions. Detection wants pixel-level labels, and we did not have those and could not make them. Classification wants an image and a word, which is what every public corpus of sick plants actually contains. It also degrades honestly: a classifier that is unsure returns a low top score, and a low top score is something you can refuse to act on. That refusal turned out to be the most important feature in the system.

The labels were the problem, not the images

The first model plateaued at about 40% on a small held-out bench, and every instinct said more images. We tried the opposite first: re-label what we already had with a second opinion we trusted more than the original tags. The second opinion agreed with the existing labels 36% of the time. The model had been learning from a corpus that was wrong roughly two images in three.

Retrained on the cleaner labels, a set one-tenth the size beat the full corpus. Then the classes the corpus simply did not contain — pests, light burn — were seeded with a few dozen examples each, and the bench went from the fifties to the eighties in one step with no change of method. The plateau had never been about volume. It was missing classes and bad labels, in that order.

Two habits came out of that phase that we would recommend to anyone doing this on a small bench:

• Average over three seeds, always. On a bench of a few dozen images, one run wobbles by an image per class. A single lucky run reported 35% on the hardest axis; the three-seed number was 25%. We had already written the 35% down.
• Only keep a label when two independent sources agree. For the nutrient classes we scored every image blind and kept only those where the blind read and the source’s own tag matched. Fewer images, much higher precision, and the hardest metric nearly doubled.

Some classes cannot be told apart, and the model should say so

The agreement matrix from that filter told us something about the problem rather than the model. Nitrogen, magnesium, iron and sulphur separate cleanly. Calcium, potassium and phosphorus are visually entangled — an expert’s calcium page reads as magnesium or potassium to a blind observer, repeatedly, and it is not a labelling error. Late-stage leaf damage converges; once tissue has died, most causes look alike.

So the product does not pretend otherwise. The classifier reports a deficiency family — uniform chlorosis, interveinal chlorosis, margin necrosis — and names a single element only when the top two scores are far enough apart and in different families. We score ourselves on three tiers: broad category, right family, exact element. On the bench those came out at roughly 90 / 82 / 53. The middle number is the honest headline. The last one is a physics ceiling as much as a model ceiling, and a bigger model will not move it much.

Colour is the signal, so you cannot augment it away

Grow lights are not white. We simulated the colour cast of common fixtures and re-ran the bench under each. Broad category held up (81–94%). Element identification collapsed — worst under amber, where the family metric fell from 82 to 47, because amber erases exactly the hue differences that separate one nutrient from another.

The standard fix is to augment the training set with colour casts so the model becomes invariant to them. We tried it. It traded 18 points of clean accuracy for essentially nothing under cast. Nutrient identification is irreducibly colour-based; teaching the model to ignore colour teaches it to ignore the symptom. What worked was normalising the illuminant at the input instead — a bounded grey-world white balance on the frame the model sees, gain clamped so a single-leaf close-up cannot be greyed out — which recovered the clean numbers under most casts with no retraining. We should say that a diagonal cast corrected by a diagonal white balance is close to a tautology in simulation, and that the real-tent efficacy is lower than the synthetic recovery suggests.

The bench did not transfer

The first time we ran the pipeline on real photographs of real plants — the founder’s own, all healthy — it returned one clean healthy, three inconclusive, one borderline stress call, and one confident case of botrytis on a seedling that had wood-chip mulch in the frame. Brown mulch reads as rot. The model had never seen soil, mulch, a tent wall or a whole plant in daylight, because the corpus is close-up leaf photography and a grower’s camera does not take close-up leaf photography.

Part of that is fixable with a gate, and we built one: a small recall-first network that rejects frames which are not the species we trained on, or are mostly surface, before the specialist ever sees them. It cuts off-species and surface frames reaching the classifier by about 99.7%. But it correctly passes a whole cannabis plant in a busy scene, and that is the frame a fixed tent camera produces every hour.

On those frames — the ones the device actually captures — we adjudicated every positive call the classifier had made over several weeks against what was really in the picture. It was wrong in both directions: it missed a real chlorosis episode most of the time it was visible, and most of what it did flag was not there. Confidence did not discriminate. The highest-confidence deficiency verdict the device has ever issued was wrong, and there is no threshold that keeps the true positives while dropping the false ones, because the wrong calls score in the same band. Failures also came in bursts: it would lock onto a plant state and repeat the same wrong call for hours.

We tried the obvious levers. Re-framing and cropping the pod frame toward the canopy: no gain, twice. Masking the background: worse. Retraining on the small number of pod frames we had: no movement. Every one pointed at the same cause. This is a data-distribution problem wearing a model-accuracy costume, and no amount of work on the model side of that gap closes it.

What the device does now

We decided that turning diagnosis off would be an admission of failure, and that the answer was to get it right instead. Getting it right, for now, means being precise about what the pipeline can vouch for.

A binary health gate runs first, trained on the frames from this specific installation. On held-out frames it reads a healthy plant as healthy every time we have tested it, including pale new growth that an earlier model called a deficiency. When it is not sure, the device says worth your eyes and shows you the frame, rather than naming a problem. The element classifier still runs on every frame — in shadow. Its verdict is recorded, never shown, so that when a human does adjudicate an episode there is a labelled example of what the model thought. That is the feedback loop, and it is installation-specific by design: the gate tuned on one tent reads every frame from a different academic set as healthy, so it must never be trusted on a tent it has not seen. Every unit cold-starts.

A separate finding settled who is in charge. We benchmarked a small vision-language model on the NPU as a free-form diagnostician. It produced fluent, plausible, cannabis-aware prose and confidently misdiagnosed a calcium deficiency as powdery mildew on an image the structured classifier had scored correctly at 0.998. The classifier is the authority. The language model turns a verdict and a family cue into grower prose, grounded on that verdict, and is never asked what it thinks is wrong.

Three bugs that were not in the model

Channel order. An entire day of laptop-side evaluation was run on images with red and blue swapped — one library treats a bare array as BGR and the image as RGB, and we passed the array. The element model’s real held-out accuracy was 80%, not the 62% we had reported, and a “calibration head” that appeared to add five points was partly correcting the swap. We found it because the converted on-device model refused to reproduce the laptop numbers. When a reference and a conversion disagree, suspect the reference’s preprocessing before you suspect the conversion.

Quantization. The NPU converter defaults to eight-bit weights and activations. On the small gate networks that is not a quality loss, it is signal destruction: one gate went from an area under the ROC curve of 0.94 in float to 0.56 quantized — a coin flip — with a third of verdicts flipping. No threshold recovers it. Every gate ships at fp16, at 1.6× the file size, and every converted model now goes through a parity harness against the float original on the same few hundred frames before it is trusted.

Preprocessing in the parity harness itself. The first version of that harness squished images to the model’s input size; the gates were trained on a resize-and-centre-crop. The harness reported a conversion loss that was actually its own resize. A checker has to reproduce the pipeline it checks.

The one that failed held-out last week

We keep trying cheaper signals than a classifier. The latest was a numeric score for yellowing at leaf margins, built from hue and local density on the pod’s own frames, with twenty-five synthetic controls that all passed. On twenty-four real crops chosen by eye from the archive it came out inverted: clean juvenile growth scored higher than visibly yellow tips.

The reason, measured rather than reasoned: on this camera, hue is not illumination-invariant. Lit lamina reads one hue, a stop darker reads ten degrees away, shadow further still — and a leaf serration tilted toward the lamp reads about as yellow as a genuinely yellow tip. Brightness matching did not remove it. The confound is edge geometry against the light, which is a polariser problem and not a software one. The module is committed with its overlays and its negative result so nobody rebuilds it from the green controls.

What we would tell someone starting this

• Fix labels before adding images. Score your existing labels with a second source and look at the agreement number before you scrape anything.
• Three seeds, or the number is not real. A small bench lies by one image per class per run.
• Find the classes that cannot be separated and report the group. A model that names an exact element it cannot see is a liability; a model that says “one of these three” is useful.
• Never train colour-invariance into a colour-based task. Normalise the input instead.
• Evaluate on the frames the device will actually see, not the frames the corpus contains. If those differ, every bench number you have is about the corpus.
• Confidence is not calibration. Before you set a threshold, check whether the wrong calls score lower than the right ones. Ours did not.
• Abstain loudly rather than widen the fit. Held-out generalisation is the only test. When it fails, say so on the screen.
• Check channel order and quantization before you trust any evaluation number. Both cost us a day and both looked like model results.

If you grow and you have wondered what a device like this can honestly tell you from a photograph, the short version is: whether a frame is worth your attention, reliably; which family of problem it might be, often; which exact nutrient, not yet. The longer version is our note on pale new growth and the episode where the device kept every reading in range and still missed the plant.

Sources

Bench figures are our own from June 2026, three-seed averages on a small held-out set, re-confirmed through the production HTTP-to-NPU path on an Orange Pi 5B (RK3588S) at fp16. The colour-cast study is a simulation (diagonal white-balance model) and is labelled as such above. The real-photograph test is from 2026-06-24, the pod-frame adjudication from August 2026, the channel-order and quantization findings from July and August, and the margin-score result from 2026-09-03. No corpus sources, sizes or labelling details are given here, deliberately. Nothing in this post is a claim about what the shipped device can find in a frame; the one figure that looked like one is the figure that did not transfer.