3D LiDAR perception: point clouds, sensor fusion and edge AI

What a LiDAR perception stack actually does

A LiDAR sensor returns a sparse, unordered set of 3D points with intensity — typically tens to hundreds of thousands per sweep, at 10–20 Hz. A perception stack turns that raw geometry into decisions: what is here, where is it, where is it going, and how confident are we.

The pipeline is usually five stages: pre-processing (motion compensation, cropping, downsampling), ground and background separation, point-cloud segmentation or clustering, 3D object detection and classification, then multi-object tracking with state estimation across frames. Each stage constrains the next — an over-aggressive downsample destroys the small-object recall your detector needs, and a noisy ground plane creates phantom obstacles that tracking then smooths into confident nonsense.

Point-cloud segmentation: geometry before learning

Learned segmentation is powerful, but classical geometry remains the cheapest accuracy you can buy. RANSAC or grid-based ground fitting, voxel downsampling and Euclidean clustering resolve a large share of scenes at a fraction of the compute of a neural network, and they fail in predictable ways — which matters when you have to explain a failure.

Where learning earns its place is semantics and ambiguity: separating a pedestrian from a post, a trailer from a wall, vegetation from a hard obstacle. Voxel and pillar encodings (sparse convolution over a voxel grid, or vertical pillars projected to a pseudo-image) dominate in production because they map cleanly onto the parallel hardware you actually have, while raw point-wise architectures are often better on benchmarks than they are on a power budget.

3D object detection under a real compute budget

The honest constraint in 3D object detection is not accuracy, it is latency at fixed power. A detector that takes 180 ms on a workstation GPU is not a detector on an embedded module drawing 15 W. The design choices that follow — quantisation to INT8, pruning, resolution and range trade-offs, restricting the detection volume to the region that actually matters — are engineering decisions with accuracy consequences you must measure, not assume.

Evaluate on the distribution you deploy into. Benchmark datasets are recorded with specific sensors, mounting heights and geographies; a model tuned on them will quietly degrade when the beam pattern, the mounting angle or the weather changes. Range-stratified metrics (near / mid / far) expose failures that a single mAP number hides completely.

Sensor fusion: when a second modality helps and when it hurts

LiDAR gives metric geometry and works in darkness; cameras give texture, colour and classification cues at far higher angular resolution; radar gives direct radial velocity and survives fog and heavy rain. Sensor fusion exists because no single one of those covers the failure modes of the others.

Fusion happens at three levels. Early fusion decorates points with image features before detection — highest ceiling, and the most sensitive to calibration and time synchronisation error. Mid-level fusion combines feature maps in a shared frame such as a bird's-eye view. Late fusion associates independent per-sensor detections and is the most robust: a sensor that drops out degrades the output rather than corrupting it.

The common mistake is treating fusion as an accuracy upgrade rather than a redundancy strategy. If the fused output cannot survive one modality failing, fusion has added attack surface, calibration debt and latency without adding safety.

Edge AI inference: shipping it, not demoing it

Edge deployment is where perception research meets reality. Getting a model onto an embedded accelerator means graph export, operator coverage checks, quantisation with a calibration set drawn from real deployment data, and a hard look at every non-network cost — point-cloud pre-processing, memory copies, and the tracker — because those frequently exceed the inference time itself.

Budget end to end, from sensor timestamp to published decision, and measure the tail latency rather than the mean. A p99 that misses the sweep period means a dropped frame at exactly the moment the scene is busy. Then instrument in production: log input statistics, detection-count distributions and confidence histograms, so drift shows up as a trend rather than as an incident.

What is LiDAR perception?

LiDAR perception is the process of turning raw 3D point clouds from a LiDAR sensor into usable scene understanding — separating ground from obstacles, segmenting and clustering points, detecting and classifying 3D objects, and tracking them over time with an estimate of confidence.

How does point-cloud segmentation differ from 3D object detection?

Segmentation assigns a label or cluster identity to points, answering which points belong together or to which class. 3D object detection outputs discrete objects as oriented 3D bounding boxes with a class and a confidence score. Many production stacks use geometric segmentation to reduce the search space, then run a learned detector on the remainder.

Is LiDAR still needed if cameras and radar are available?

LiDAR provides direct metric geometry independent of ambient light, which cameras cannot, and far higher spatial resolution than radar. The practical argument for keeping it is redundancy: fusing LiDAR, camera and radar means one modality's failure mode — darkness, fog, textureless surfaces — is covered by another.

What limits LiDAR perception on embedded hardware?

Latency at fixed power, not raw accuracy. Point-cloud pre-processing, memory bandwidth and tracking often cost as much as the neural network itself, so the useful metrics are end-to-end tail latency against the sensor sweep period and range-stratified accuracy after quantisation.

Which metrics matter when evaluating a 3D detector?

Range-stratified precision and recall rather than a single aggregate mAP, orientation error for oriented boxes, tracking identity switches over time, and calibration of the confidence scores. Aggregate benchmark numbers hide the near-range and far-range failures that determine real-world behaviour.