University of Alicante

Researchers at Institute for Computing Research of the University of Alicante have developed a new system capable of providing a natural interaction to disabled performing motor and cognitive rehabilitation therapies. In this way, the system offers a new approach making an efficient use of new technologies and man-machine interaction sensors.

The system combines the use of various sensors. A long range sensor (such a 3D camera) that enables user identification and interaction with the system through gestures or patient own pose. This sensor also provides depth information of the scene, enabling to locate precisely the patient operation environment.

The data obtained by the long-range sensor can be combined with the use of another short-range 3D sensor that provides a precise virtual representation of users’ hands. This information is used and combined with information from other sensors to perform rehabilitation therapies requiring precise movements of hands, a pose or certain fingers gesture. This sensor is located on a swivel arm, enabling its use when needed.

Also, the system incorporates a gaze tracking sensor capable of obtaining the patient’s gaze position (eye tracking). This sensor is calibrated for each patient to find out the position of the patient's eyes regarding user interfaces displayed on the tablet screen.

The sensors mentioned are combined with a microphone array, enabling to capture users’ voice commands, and sound information of the environment.

Finally, all sensors are connected to a digital tablet that hosts the information processing and offers other means of additional interaction: interaction by contact. This mode of interaction also enables rehabilitation therapies or communication based on patient contact with the interfaces displayed on the tablet screen.

All components could be mounted on a mobile platform forming a complete system of natural interaction prepared to perform therapies with patients with different disabilities.

Regarding the mode of operation, the system has a working environment for developing applications that enables rehabilitation or user communication through various human-machine interfaces.

Initially, the system identifies a user in the environment leaving settings adjusted to patient’s requirements and characteristics according to their disability. This task can be carried out with a therapist or caregiver assistance who registers the user and trains the system for that patient.

The system receives gestural data as input and translates it to basic commands (up, down, left, right, etc.). In addition, the system enables patient’s own gestures customization for each patient and sensor. Thereby, the creation of a different set of poses, gestures with the body, arms, head, hand, eye movement and precise locally hand movements (finger movement) or voice commands are possible to carry out the same actions on the system. Through these actions the user can navigate the environment, options, applications, etc., to perform rehabilitation or specific communication with a specific application.

Figure 1.- Frontal view of prototipe

Each patient can be recognized in the system using biometric identification (facial recognition) and can also define a set of sensors and a set of custom gestures to be used depending on the type of disability. Through this customization and the proper sensors calibration, the system ensures the best user adaptation offering the most natural interface.

The system includes a set of applications enabling both interaction and patient rehabilitation. In particular, several applications taking the basic commands as input have been developed: predictive text application (convert text to speech); application motor rehabilitation (through virtual object manipulation) and cognitive rehabilitation (through guided drawing). However, this initial set of applications is extensible and adaptable according to the different needs of patients. In this way, the system provides a flexible and modular workspace for new therapies development.

• Enables interaction and rehabilitation therapies performed to disabled with both motor and cognitive impairment.

• Enables multiple modes of interaction (gestures, poses / hand movement, voice, look, and touch) for people with different disabilities offering significant advantages over other systems.

• Enables customization of patient own gestures for each sensor providing a natural interaction experience with the system.

• Provides biometric identification (facial recognition) patients adapting the interaction (sensors and gestures) depending on the user disability level.

• Combines the data obtained from the sensors with 3D interfaces efficiently. The system provides a more realistic way of rehabilitation through the use of advanced virtual reality techniques.

• It provides a flexible and modular development workspace of new applications oriented to new therapies based on the different needs of patients.

• Comprises sensors and devices available on the market, therefore, can be modified, adapted and replicated easily at a reasonable cost depending on the type of patient, disabilities and therapies to implement.

The system has been developed and successfully tested at laboratory. A prototype (see Figure 2) with the following characteristics is available:

• The prototype includes sensors to capture information from hand, eyes and body (head, arms, etc.) movements. The system is open and is ready to incorporate new sensors. Currently, the research group is performing new tests to include bracelet-type biorhythms sensors. They have also fully developed the platform where the sensors and tablet are placed.

• The group has developed an application of cognitive rehabilitation based on image cards. The user must select one of the displayed images. Another application developed is the search for the exit in a maze.

• The user management part of the system (identification by biometrics, selection of devices adapted to disability, and training and recognition of user gestures) is already fully developed.

Figure 2.- Side view of prototype

The proposed system can be used to perform rehabilitation therapies for people with motor and cognitive impairments. The system also facilitates communication with others through a customized and natural interaction between patients and devices (sensors, tablet, etc.).

Potential stakeholders:

• Companies for disabled support systems development and trading.

• Associations or public/private entities aimed at providing support services to disabled.

• Regional or national governments in the Social Welfare and Disability areas.

Companies and organizations interested in technology use and/or commercial exploitation are sought. Possible agreements:

• License agreements (use, manufacture or marketing).

• R & D project agreement (technical cooperation) for system use and / or adaptation to other sectors.

• Subcontracting agreement (technical assistance, turnkey solutions, training, etc.).

This technology is protected by patent:

• Application number: 201531430

• Application date: 05/10/2015