Improved LiDAR distance measurement with embedded AI

Both in autonomous driving and in industry, disturbances from the environment, such as background light, are always a challenge for LiDAR systems. For the reliable use of optical sensors, it is crucial how disturbances from environmental factors can be reduced to improve the robustness of the system while maintaining real-time capability. In a research project, Fraunhofer IMS is developing a robust machine learning-based algorithm for distance determination called "multi-peak analysis (MPA)" for LiDAR histogram data under harsh detection conditions. Rough detection conditions in this context include environments with strong background light (high background photon rate), situations with objects at long distances, and a high required frame rate.

A variant of MPA is the neural network based MPA (NNMPA), which uses a very compact feed-forward neural network without compromising the precision of the detected distances. After a physically driven feature extraction, the neural network requires only 1% of the original data for training and prediction. For the evaluation of the NNMPA, 100,000 LiDAR histogram data (both synthetic and real) based on the flash LiDAR system "OWL" developed by Fraunhofer IMS are used.

NNMPA was compared to classical digital processing, which includes a mean filter, background subtraction, and global peak detection (AF-BS-GPD), and shows superior results. For example, at target distances of 25 to 60 m and high background light levels, the distance determination accuracy of NNMPA is 26% higher than that of AF-BS-GPD. An example can be seen in Figure 1. In this particular scenario, the AF-BS-GPD is strongly perturbed by background light (left diagram). This results in almost half of the distance determinations being incorrect. The measured distances should be much smaller than they actually are (clustering of the "red dots" in the lower region). The NNMPA (right diagram), on the other hand, achieves an overall accuracy of almost 88%. Furthermore, the NNMPA additionally provides the information of the measurement quality, which is represented by the color in the right diagram. Furthermore, it can be seen from the right diagram that the false determination results have a relatively low measurement quality. This information is particularly useful for subsequent further processing and decision-making.

Comparison of LiDAR sensor data without and with AI
© Fraunhofer IMS
Comparison between AF-BS-GPD and NNMPA. The top view of the simulated scenario with objects at a distance of 25 m and a wall in the background at a distance of 35 m is shown. The background photon rate is 5 MHz. The points shown in color indicate the prediction result of the corresponding processing method.

The fixed-point version of the NNMPA has been successfully implemented on a small FPGA evaluation board with an accuracy drop of only 1.2% compared to a floating-point version. Furthermore, a runtime demonstration in LabVIEW shows that the NNMPA requires less computation time and thus achieves a higher frame rate than the AF-BS-GPD. In addition, experiments show that the NNMPA can learn background light information on its own instead of applying conventional noise filtering, which is usually very computationally intensive.

Currently, Fraunhofer IMS is developing a correlation algorithm between neighbor pixels and time frames based on NNMPA. This algorithm is expected to take the reliability of distance prediction to a new level. Further concrete applications with the NNMPA are planned for the future.


Faster and data reduced solution for high-resolution LiDAR-systems: TimestampsAI is our latest development for your applications to capture complex scenes in 3D in any environment.

LiDAR Camera Owl

192 pixel x 2 lines SPAD based camera

Publication of the Fraunhofer IMS

Data Processing Approaches on SPAD-Based d-TOF LiDAR Systems: A Review

Publication of the Fraunhofer IMS

Feature Extraction and Neural Network-based Analysis on Time-correlated LiDAR Histograms

Our technologies - Innovations for your products

Distributed Learning

Distributed learning enables training of complex tasks on multiple small embedded systems.

Hybrid Learning and PGNNs

In case of insufficient training data simulations based on physical models help to improve the data base.

Feature Extraction for Smart Sensors

By means of adapted feature extraction the size of a neural network can be reduced.

Industry 4.0 Research

Against the background of "Industry 4.0", Fraunhofer IMS is researching predictive maintenance solutions for the manufacturing industry.

Our technology areas - Our technologies for your development

Communication and Networking

Communication interfaces allow data exchange with other devices and connection to networks

User Interfaces

User interfaces as interface between device and user allow the configuration and operation of a product

Machine Learning for Embedded Systems

Artificial intelligence on resource-limited systems can be used to extract higher quality information from raw sensor data

Computer Vision

Computer Vision methods extract the maximum amount of information from image data


Embedded Software and Artificial Intelligence (Home)

Here you can get back to the overview page of the core competence Embedded Software and Artificial Intelligence (ESA).