Intuitive operation of devices through acousto-gestural control

Sometimes it is necessary to control devices or technical systems without using hands. The reasons for this can be very different. Either the hands are needed for other things (e.g. for the repair or maintenance of a system) or they are simply not available due to an illness or physical limitation. Instead of conventional input media such as a keyboard or voice input gesture control is a good alternative option here.

However, gesture control only covers the communication channel from human to machine. For the return channel from the machine to the human displays are used, for example. Another possibility is an acoustic interface where instructions or status messages are spoken or read aloud and picked up by the human ear. We call this combination of gesture input and acoustic output “acousto-gestural control”.

Fraunhofer IMS has developed an acousto-gestural control interface which is integrated into a headphone to operate the medical device EQUIVert® without classical control elements only with the head.

The sequence control is performed by a commercially available microcontroller with a connected 9-axis Inertial Measurement Unit (IMU) for capturing the movements of the head. The head acceleration data is captured by the sensor and processed with a patented method in such a way that only a few features from the real-time signal have to be forwarded to a neural network which then performs the classification of the head movement such as nodding and shaking. With the recognized gestures the user of the device can then navigate through the menu items of the system, configure the device and control the training process.

The embedded AI library "AIfES" (Artificial Intelligence for Embedded Systems) developed by Fraunhofer IMS is used for the realization which allows Artificial Neural Networks (KNN) to be executed and trained on microcontroller platforms with extremely limited resources (computing power, memory, energy consumption).

The text messages for the control of the medical device are digitized in the memory, in the case of EQUIVert® on a SD card, and are forwarded by the sequence controller via direct memory access methods to a digital-to-analog converter with a downstream audio amplifier. In the case of EQUIVert® these are spoken text fragments on the menu items and settings of the system as well as on the instructions for balance training. The text fragments are then automatically assembled by the system as needed and played back to the user. Feedback from the user is then provided via a corresponding head gesture.

The acousto-gestural control offers the user of EQUIVert® an intuitive user control that does not require any further operating elements and ensures that the user's hands are always free to safely perform balance training.

In addition to this medical application the acousto-gestural control is also suitable for other applications such as gaming applications, expert systems for technicians or training systems in the field of rehabilitation.