For rescue missions it is among other things important to locate missing persons as quickly as possible, e.g. in imponderable terrain or in smoke-filled buildings. Another application of person detection is in security technology where the presence of persons in restricted areas must be detected. In principle, classic video cameras can also be used for these tasks. However, these fail in difficult visibility conditions: At night, additional spotlights must be used as fog or smoke prevent classic video cameras from acquiring images and glare such as low sun or backlight overrides image acquisition. These disadvantages become obsolete through the use of thermal imaging cameras. Thermal imaging cameras use the radiation power emitted by objects in the long-wave infrared spectral range (LWIR with a wavelength of 8 µm to 14 µm). This makes it a passive image acquisition, because all objects with a temperature greater than 0 K emit electro-magnetic radiation whose maximum intensity at body temperature is about 10 µm wavelength. Therefore, no active illumination of the scene, e.g. by spotlights or by the sun, is required. Due to its longer wavelength than visible light, LWIR radiation also penetrates fog and smoke.
Artificial neural networks, specifically convolutional neural networks, are typically used to classify people in video images. The necessary algorithms require high-performance systems to perform object detection in real time. Fraunhofer IMS has developed artificial neural networks for person recognition, especially for resource-constrained systems. With these, a recognition rate of 83% and a parameter reduction of more than 99% could be achieved compared to state-of-the-art methods of object recognition. This enables the use of person recognition on low-power microcontrollers with a clock frequency of 160 MHz and a 4 MB RAM in real time. This means that no connection to a cloud or similar connection via Internet is required, as the algorithm is fully integrated and executed autonomously on an embedded system. Thus, we are able to reliably detect and mark persons in thermal images in an automated way.
The thermal imaging camera used is an in-house development of the Fraunhofer IMS, which is based on a digital 17 µm QVGA IRFPA (also an in-house development) as a thermal image sensor for LWIR radiation. In the two thermal images, the respective recognized persons are marked by means of a rectangle. With a high recognition rate of 83 %, persons can be recognized in thermal images. By reducing the parameters by 99 %, the display software of the IMS thermal imaging camera, including the algorithms for person recognition, on a Raspberry Pi, achieves a real-time capable refresh rate of 20 Hz.
The display software shows a large false-color image of an example scene. In this scene two persons were detected and marked by rectangles. In the upper right corner the coordinates of the recognized persons are displayed. These can be sent, for example, via a low-power wide-area network if an Internet connection is not possible and thus a transmission of the image data is not possible. At the bottom right is a grayscale image of the scene.
The results presented were produced as part of a Fraunhofer project "RuLe" ("Resilience of Urban Living Spaces") in which the Fraunhofer Institutes FHR, IMS, LBF, IML, FKIE, INT and IAO cooperated together. The work on the aspect "embedded AI" was carried out by Fraunhofer IMS on its own.