Anyone who lives, in some way, at work or at home, with a hearing impaired person faces difficulties on a daily basis when trying to initiate communication with them through an act or gesture. This is the theme of a video that humorously shows a young deaf or hearing impaired person in his home, focused on reading a book, in a position that does not allow him to see the other occupants of the space and those who may pass by. When they want to attract your attention to start communication, they use the classic resources of direct visual contact, physical touch, light signals such as turning lights on and off, knocking on the ground or furniture in order to produce vibrations that can be perceived. The laziest throw pillows that are at hand. 

For decades, the methods used to capture the attention of a hearing-impaired or deaf person to initiate an interaction have always been the same. Despite the profusion of communication technologies in use among the hearing impaired and deaf communities, the difficulty in capturing their initial attention has not yet been overcome with the use of new technologies.

This scenario motivated Marcelo Sodré Plachevski, a graduate in IT Technology, to dedicate himself to his master's degree in which he designed and developed a system that uses voice recognition technology, through a mobile device, sensitive to the voice of any speaker, capable of generate a vibrating and luminous alert for the deaf person when one of the words previously registered in the system's dictionary is pronounced. The work was supervised by professor Rangel Arthur, from the Telecommunications Division of the Faculty of Technology at Unicamp, Limeira campus, current coordinator of the Telecommunications Engineering course, and was co-supervised by professor Francisco J. Arnold. 

The researcher remembers that there is no technology with voice recognition aimed at alerting the hearing impaired as proposed in his project. Even today it relies heavily on traditionally used alerts. For this reason, the idea that led to the work constitutes an innovation and aroused the interest of the National Federation of Education and Integration of the Deaf (Feneis) of São Paulo, when the initial concept of the idea was presented by him while still in the conception phase.

Still at the beginning of the development of the study, Marcelo discovered this reality when visiting the International Fair of Technologies in Rehabilitation, Inclusion and Accessibility (ReaTech), which takes place annually in São Paulo and brings together all existing technologies aimed at disabilities in general. At the time, when maintaining contact with associations for the hearing impaired and with companies that work with products aimed at hearing, he confirmed that there was nothing similar to what he intended to create. 

The idea

In general terms, the idea that guided the construction of the system created by the researcher is simple. The hearing impaired person keeps, in front of them or close to their body, a smartphone that emits a light signal or vibrates when a person nearby pronounces their name. The disabled person becomes aware that their attention is being requested and then looks at the speaker with a view to establishing communication that can develop with the use of various resources, including the use of Brazilian Sign Language (Libras). Tests showed that the system works at distances of up to five meters, indoors and outdoors, and supports up to certain noise levels, as the quieter the environment, the greater the possibility of adequate recognition.

The importance of the work is highlighted when it is known that the world population with some type of disability is estimated at one billion people, according to the World Health Organization (WHO). In Brazil, according to the 2010 IBGE census, 45,6 million people have some type of disability, which represents 23,9% of the population. Of this universe, 9,71 million declared themselves to have some type of hearing impairment. Given this situation, the author of the dissertation initially asked himself what could be done to help these people and how technology can be used to make their lives easier. Regarding the importance of these initiatives, he found support in what Mary Pat Radabaugh, an expert in the field of Assistive Technology, says: “For people without disabilities, technology makes things easier. For people with disabilities, technology makes things possible.”

Assistive Technology constitutes an interdisciplinary area of ​​knowledge that encompasses products, methodologies, strategies and services with the aim of promoting the autonomy and independence of people with disabilities or special needs, providing them with a better quality of life.

Based on this concept, Marcelo created the Assistive Technology system for Aid to the Hearing Impaired and Deaf (TAADA), designed to eliminate the gap in the process of capturing the initial attention of the hearing impaired or deaf person from a person considered hearing.

The system

Using voice recognition technology and selected words, in conjunction with a mobile device, the smartphone, the TAADA system acts as an interface between the hearing impaired and the listener. TAADA makes it possible to draw the attention of the disabled person through sound alerts emitted by the smartphone or light signals on the device's display whenever someone calls the hearing impaired person by the name previously registered in the system. The author details that TAADA uses voice and word recognition resources from pocket sphinx, Through a front end that works with it, aggregated into a smartphone with the Android operating system. This set allows the spoken word to be recognized regardless of the speaker.

The developed system is composed of physical and logical items. The physical layer, also called hardware, consists of a smartphone that has the ability to vibracal. The logical layer contains the Android operating system, the speech recognition system pocket sphinx, data dictionary and an application interface for user use.

The author remembers that free software pocket sphinx required changes and adjustments necessary to be used in speech recognition so that it could perform the intended functions. In the original version the pocket sphinx transforms speech into text, which works very well for the English language, but does not meet the needs of the project. Based on a table listing the most common proper names, tests were carried out to test the system's efficiency. “It is worth highlighting that our software is free, open source, and will soon be available on the internet”, recalls Rangel.

The teacher draws attention to the fact that most voice recognition software on the market requires training with the speaker to convert the voice signal into text. As it is difficult to find software that identifies any voice, this was the initial difficulty to be overcome and which led to restricting the number of words to be used, but which could be pronounced by any speaker both indoors and outdoors, as long as the noise does not exceed a certain number of decibels.

Marcelo explains that, having defined the voice recognition process and the use of the smartphone as hardware, he began to study the possible delimitations of the project. Therefore, efforts were made to ensure that the system was capable of recognizing what was said regardless of who said it, eliminating the need for speaker training and, furthermore, that it could work without the smartphone being connected to the internet, thus ensuring greater universalization of its use. The use of smartphones also resulted in a reduction in the number of words to be recognized. In this case, a list of proper names was used, considering that they are what the disabled person prefers to be called. The system captures the voice of any speaker, identifies the name called and previously registered, and issues an alert signal to the disabled person. No hearing impaired person will receive cushions from family members who ask for attention.

Publication

Dissertation: “Assistive technology system to capture the attention of hearing impaired and deaf people”
Author Marcelo Sodré Plachevski
Advisor: Rangel Arthur
Co-advisor: Francisco J. Arnold
Unity: Faculty of Technology (FT) at Unicamp/Limeira