Handwriting recognition by ANN – enabled by AIfES

Nowadays, there are a wide variety of interfaces and methods that can be used to capture and recognize handwriting. With the help of the AIfES framework (Artificial Intelligence for Embedded Systems) developed by Fraunhofer IMS it is now possible to have a microcontroller recognize handwriting entered on a touchpad. Normally, handwriting recognition is a computationally intensive task that must be performed by a powerful PC. By using the extremely resource-saving AIfES library it is now possible to have such operations performed even on small microcontrollers or on the touchpad itself.

The character entered manually on the touchpad is captured in the form of the raw data generated. From this raw data, features are extracted by a software using an innovative patent-pending process which are then evaluated by a trained neural network. The training of the neural network is also done by AIfES on the microcontroller. The touchpad only sends the results, in this case the recognized character, to the computer. This effectively reduces data volumes, minimizes runtimes and, on the other hand, increases data security.

This is facilitated by the platform independence of AIfES which allows the framework to run on almost any microcontroller. This results in many degrees of freedom for the selection of suitable hardware. Instead of the usual keypads a simple and fast way of handwriting recognition is possible which can be used especially in the areas of smart home or smart city. Examples are the input fields of ATMs, security systems, telephones or even calculators. But also, an extended touchpad function of laptops can be implemented.

A first demonstrator by Fraunhofer IMS proves the functionality of handwriting recognition by the capturing of handwritten digits. An extension to the additional recognition of letters is possible as well as the recognition of more complex gestures for operating devices which are detected by suitable sensors. Fraunhofer IMS is happy to make its expertise and technology available for the development of new applications and products.

For the experimental setup shown here a standard PS/2 touchpad was used in combination with an Arduino Uno. The results of the handwriting recognition are shown on an LC display. Under these conditions a calculation time of about 20 to 25 ms could be achieved depending on the number of recorded measuring points and the length of the character string: Since, for example, the character string of an "8" is longer than that of a "1" and thus more measuring points are recorded, the evaluation of an "8" is more complex. To train the network only 360 digits were used in the form of 36 data sets of the numbers “0” to “9” entered by different persons. The recognition rate of the training data was already over 95% with this manageable data set and can be further optimized with a larger amount of training data.

Our technologies - Innovations for your products

Gesture Recognition

In gesture recognition the embedded system uses suitable sensors to evaluate the movements specifically performed by the user in order to use this as operator input.

Acoustical Gesture Control

With this combined type of operation, the embedded system gives instructions to the user by speech output while the user controls the system by using defined gestures.

Our technology areas - Our technologies for your development

Communication and Networking

Communication interfaces allow data exchange with other devices and connection to networks

User Interfaces

User interfaces as interface between device and user allow the configuration and operation of a product

Machine Learning for Embedded Systems

Artificial intelligence on resource-limited systems can be used to extract higher quality information from raw sensor data

Computer Vision

Computer Vision methods extract the maximum amount of information from image data


Embedded Software and Artificial Intelligence (Home)

Here you can get back to the overview page of the core competence Embedded Software and Artificial Intelligence (ESA).