What 10+ clients taught me about LiDAR annotation quality

Two years of freelance dataset engineering across autonomous driving, CCTV surveillance, retail AI and sports analytics, for more than ten clients. Here's what actually determines whether a labelled dataset is worth anything.

Consistency beats accuracy

A dataset where every annotator makes the same small error is more useful than one where each makes different perfect-looking judgements.

Models learn the labelling function you gave them. If your taxonomy is ambiguous — where exactly does "road" end and "sidewalk" begin at a dropped kerb? — annotators resolve it differently, and the model learns noise at exactly the boundary you care about.

The fix is not better annotators. It's a taxonomy with no ambiguous cases, written before labelling starts, with worked examples for every edge case someone will hit.

Thing/Stuff is a real distinction

Panoptic segmentation splits classes into:

• Things — countable instances. Cars, people, cyclists. Each gets its own instance ID.
• Stuff — amorphous regions. Road, sky, vegetation. No instances, just a region.

Getting this wrong is expensive. Treat a crowd as stuff and you lose per-person tracking. Treat road markings as things and annotators spend hours drawing boundaries nobody needs.

Format compliance is not a formality

COCO, KITTI and YOLO encode boxes differently — normalised versus absolute, centre-based versus corner-based, different axis conventions for 3D.

A silent conversion bug produces a dataset that loads without error and trains to garbage. Every pipeline I build now validates a random sample by rendering annotations back onto source frames. If a human can't see correct boxes, the model can't learn them.

Multi-frame 3D tracking is where consistency dies

Single-frame 3D detection is tractable. Tracking the same object across frames is where datasets fall apart: the ID switches, the box jitters, an occluded object gets a new identity when it reappears.

Our QA pass checked ID persistence across occlusion events specifically, because that's the case models fail on and the one annotators most often get wrong.

The workflow that stuck

1. Taxonomy document with worked edge cases — before anyone labels anything
2. Pilot batch, all annotators, same 50 frames, measure disagreement
3. Resolve every disagreement into the taxonomy doc, then re-pilot
4. Production labelling with automated format validation on ingest
5. Visual QA on a random sample, rendered back onto source
6. Targeted review of occlusion and boundary cases

Steps 1–3 feel like delay. They're the only reason steps 4–6 don't turn into a rework loop.