PROSPECTS FOR USING AUGMENTED REALITY (AR) TECHNOLOGY TO FACILITATE INFORMATION PERCEPTION IN INDIVIDUALS WITH HEARING IMPAIRMENT

ABSTRACT

This article examines the potential of augmented reality (AR) technology to enhance the perception and understanding of information for individuals with hearing impairments. The authors provide a review of existing developments in this field, such as AR gadgets and headsets, augmented reality applications utilizing geospatial data and recognition, translation, and representation of sign language in digital format, as well as AR marker technology. The article proposes new approaches to improve communication and social integration of a group of people with hearing loss. In particular, special attention is paid to the possibilities of using immersive and tactile technologies to create three-dimensional and interactive models that facilitate the visualization of auditory information and provide innovative means of perception and communication. This scientific material is a review based on a primary analysis of academic sources. The results of the study are an addition to the existing scientific literature and can be useful in the development of a versatile mobile application for individuals with hearing impairment.

АННОТАЦИЯ

В статье рассматривается потенциал технологии дополненной реальности (AR) для улучшения восприятия и понимания информации людьми с нарушением слуха. Авторами проводится обзор существующих разработок в этой области, таких как: AR гаджеты и гарнитуры, приложения на основе дополненной реальности с использованием геопространственных данных и распознавания, перевода, представления языка жестов в цифровом формате, а также AR маркерная технология. В статье предлагаются новые подходы для усовершенствования коммуникации и социальной интеграции группы людей с нарушением слуха. В частности, особое внимание уделяется возможностям применения иммерсивной и тактильной технологиям, позволяющим создавать трехмерные и интерактивные модели, которые облегчают визуализацию слуховой информации и предоставляют инновационные средства восприятия и коммуникации. Данный научный материал представляет собой обзор, основанный на первичном анализе академических источников. Результаты исследования являются дополнением к существующим материалам научной литературы и могут быть полезны в области разработки многофункционального мобильного приложения для лиц с нарушением слуха.

Keywords: augmented reality, 3d technologies, hearing impairments, smart glasses, geospatial AR, AR codes, haptic technology, innovations, mobile apps.

Ключевые слова: дополненная реальность, 3d технологии, нарушения слуха, умные очки, геопространственная дополненная реальность, AR коды, тактильная технология, инновации, мобильные приложения.

Introduction. The development of technology in recent decades has led to significant progress in the creation of innovative solutions in many industries, including education, medicine, sports and many others. One of the notable accomplishments in the realm of information technology pertains to augmented reality (AR) technology. AR is a form of interaction between the real world and virtual content, where digital elements such as images, animations, videos, sounds, and text complement the real environment, creating a visual and informational space for the user. Thanks to advances in the development of technical equipment, including software and hardware components, as well as improvements in augmented reality devices (such as smartphones, glasses); AR offers new perspectives for facilitating the perception of information and reducing the limitations faced by people with hearing impairment. The main advantages of augmented reality for this group of people are the conversion of audio information into visual information, support for communication, as well as improved access to information.

This article explores the prospects for using augmented reality (AR) technology to improve information perception and comprehension in individuals with hearing impairment. The initial segment of the article, relying on a literature review, the authors examine current advancements in this domain, encompassing AR devices, augmented reality-based applications employing geospatial data, sign language recognition and translation into digital formats, along with AR marker technology.

The subsequent section of the article introduces innovative methods aimed at enhancing communication and social integration among individuals with hearing impairments. Special emphasis is placed on leveraging diverse immersive and tactile technologies to facilitate the production of interactive, haptic models, thereby aiding in the visualization of auditory information and offering groundbreaking avenues for communication.

The concluding segment of the investigation contributes supplementary findings to the extant body of scientific literature, consequently holding significance for forthcoming research endeavors aimed at designing a feature-rich mobile application that is specifically adapted for use by people with hearing loss. Given the pervasive ubiquity and prevalent adoption of mobile devices by users, the utilization of mobile applications presents an optimal and convenient solution, obviating the necessity for bearing and transporting an abundance of accessories and devices. By capitalizing on these results, developers can forge innovative applications that unlock novel communication avenues and foster social integration for users grappling with such disabilities.

Statistical Findings

Currently, the prevailing and widely accessible framework for delivering technical assistance and functionality in relation to the implementation and utilization of augmented reality technology is primarily centered on smartphones. According to current statistics, as of 2022, smartphone owners used about 6.5 billion smartphone subscriptions (approximately 68% of the world's population), including the purchase of more than one smartphone by a user. This figure is expected to grow to almost eight billion by 2028 [1]. This is due to the increased availability and low cost of smartphones, as well as their versatility and a wide range of functionalities. As mentioned earlier, in addition to information sharing and communication, this indispensable attribute has sufficient capabilities to integrate with augmented reality (AR).

Based on global statistics, by 2023 the number of potential mobile user devices with augmented reality will reach 1.4 billion. Market growth is expected to span both the corporate and consumer sectors, including augmented reality digital capabilities [2]. It is noteworthy that the affordability of smartphones and their built-in augmented reality capabilities create an ideal symbiosis, enabling individuals with hearing impairments to engage with their environment at a heightened level.

According to estimates by the World Health Organization (WHO), it is projected that over 5% of the global population, equivalent to approximately 430 million individuals, necessitate rehabilitation services to address the problem of incapacitating hearing impairment (432 million adults and 34 million children) [3]. Hearing loss refers to the degeneration of auditory perception and comprehension, manifesting as either total loss of hearing (deafness) or partial impairment (hard of hearing). One potential application of augmented reality is the development of specialized devices or applications that assist persons with hearing impairments in not only enhancing their ability to perceive information but also in better understanding and interpreting the sounds in their surroundings. These innovative solutions may include various features such as speech transcription, translation of audio signals into a text message or into a sign language animation.

Although conventional approaches like sign language and captioning have played a vital role in facilitating communication, there remains space for further enhancement and innovation. Through the integration of visual prompts, captions, video clips, and real-time translations into the user's environment, augmented reality (AR) has the potential to enhance the accessibility and inclusivity of diverse communication settings, including public announcements, live broadcasts, and cultural and educational events. This innovative approach holds the promise of narrowing the divide between individuals with hearing impairments and the auditory world, thereby opening up novel avenues for effective communication and learning.

Research methodology

The research methodology employed in this study entails performing a review of the scholarly literature. Primarily, the study was formulated with a descriptive objective, centering on the exploration of prevailing technologies elucidated in prior scholarly publications. The literature review examined various technologies, studies, and conceptual approaches to provide a more complete and in-depth picture of the research area.

The principle of operation of augmented reality

Augmented Reality (AR) integrates the real world and virtual objects, creating a flexible and interactive environment for the user. This technology enables the superimposition and presentation of supplementary information, 3D models, or graphics onto the authentic surroundings.

The functioning of augmented reality encompasses several fundamental stages. Initially, the system scrutinizes and comprehends the surroundings with the aid of sensors, cameras, depth sensors, or GPS. Subsequently, the captured data undergoes processing to ascertain the user's position, orientation, as well as the attributes of the adjacent objects and surfaces.

At this stage, based on computer vision, image processing and object recognition algorithms, the system determines which virtual objects or information should be rendered and where they should be placed within the real world.

The system then displays these virtual objects on the display of the user's device, such as a smartphone, glasses, tablet, or special augmented reality headsets. This may be in the form of an overlay image, a 3D model or graphic, sound effects, or other type of information [4].

Results and discussion

Survey of existing technologies

AR smart glasses

Attaining comprehensive access to lecture information presents challenges for university students who are deaf or hard of hearing (DHH) and attend classes alongside hearing instructors and peers. Unlike their hearing counterparts who can simultaneously process lecture information through both auditory and visual channels, DHH students rely solely on visual input. Universities typically offer a range of accommodations to ensure the visual accessibility of spoken lectures for DHH students. These accommodations often include sign language interpreting and/or captioning services. Additionally, note-takers frequently assist by providing written summaries of lectures.

The advent of augmented reality smart glasses presents novel avenues for enhancing information perception and facilitating everyday life for individuals with hearing impairment. These advancements predominantly leverage augmented reality (AR) technologies, incorporating a fusion of transparent display-equipped frames alongside integrated microphones, cameras, and sensors.

One primary advantage of incorporating AR technology into smart glasses for individuals with hearing impairments lies in the provision of real-time information via the visual channel. Through the utilization of speech-to-text functionality, text identified by the speech recognition system can be promptly processed and presented on the display of the glasses. This functionality empowers individuals with hearing impairments to read and comprehend spoken words and sentences nearly instantaneously, thereby bestowing upon them heightened independence and improved communication capabilities [5].

When employing augmented reality (AR) to assist individuals with various communication disorders, additional technologies are necessary to compensate for the lack of specific stimuli. In the context of hearing loss, notable technologies include Automatic Speech Recognition (ASR), Text-to-Speech Synthesis (TSS), Audio-Visual Speech Recognition (AVSR), and Gesture Recognition (GR). ASR enables the identification and conversion of human speech into readable text, accessible through smartphone applications or directly integrated into head-mounted displays. Conversely, TSS converts written information into speech, proving particularly beneficial for deaf-mute individuals or those who are completely deaf and struggle to articulate understandable sounds. Efforts to bridge the auditory and visual aspects have resulted in AVSR, a system that enhances speech recognition by combining audio data with visual cues, such as facial expressions, in real-time. In the future, similar applications are expected to expand into other domains, such as exhibitions and performances, where it is typically challenging to simultaneously view objects and access text commentary. In these scenarios, augmented reality (AR) would not only present subtitles but also include sign language translation, thereby compensating for the absence of audio [6].

Sign language translation mobile application based on AR

In the last decade, significant advancements have been made in the realm of mobile applications that facilitate the translation of audio speech into sign language, utilizing augmented reality as a means of synthesis. These mobile applications offer swift and precise translation, ultimately enhancing sensory perception. The principal objective of these applications is to identify the user's spoken audio and subsequently generate corresponding gestures via 3D animation, displayed on the screen. Additionally, users have the option to select their preferred sign language, wherein the application will furnish appropriate hand movements in accordance with the chosen language [7].

An application of this nature possesses the capacity to substantially enhance the modes of communication and social engagement for individuals afflicted with auditory impairments across diverse domains of existence, encompassing education, employment, and everyday interchanges.

Geospatial AR

Geospatial Augmented Reality (GeoAR) is a technology that combines information about the physical environment with digital data based on geolocation. The combination of augmented reality (AR) and geographic information systems (GIS) brings about a distinct characteristic in which the traditional, static, indoor, and man-machine interactive mode transitions to a dynamic, outdoor mode. This integration amalgamates the virtual world of GIS with the genuine external world, resulting in the creation of an imaginary-real, man-made world. The potential capabilities of GIS and AR encompass a broad range of possibilities, including but not limited to mapping both natural and built environments to offer comprehensive access from any location worldwide, utilizing 3D mapping techniques to depict various elements such as locations, assets, terrain, utilities, and energy resources, and leveraging augmented reality to enhance data visualization through immersive experiences. [8]

GeoAR is capable of generating visual indicators and cues to facilitate the comprehension of the surrounding environment for individuals with hearing impairments. Specifically, the integration of augmented reality applications with geospatial models allows for improved orientation within unfamiliar territories. These applications are capable of presenting textual messages or video information pertaining to the nearby objects and occurrences on the screen of the respective device. Moreover, they can provide concise guidance on reaching a specific destination or following a route through the utilization of visual cues.

In general, geospatial augmented reality holds considerable potential for enhancing the quality of life among individuals with hearing impairments by equipping them with supplementary tools for environmental interaction and accessing information that may otherwise elude them due to their disability.

AR codes

An augmented reality (AR) code is a state-of-the-art technological advancement that enables users to engage with three-dimensional (3D) content in real-time within their physical environment. Externally, the AR code bears resemblance to the QR code; however, it possesses a more extensive range of functionalities. By scanning an AR code, users gain the capability to incorporate and interact with 3D content seamlessly integrated into their surroundings.

AR-codes hold the potential to function as a mechanism for delivering supplementary information to individuals experiencing hearing impairments. For instance, in museum or exhibition settings, the utilization of AR codes can exhibit historical facts or provide supplementary information directly on the screen of a smartphone or other device. This approach enables users to access pertinent information without relying on audio guides or seeking assistance from others [9].

AR codes possess the potential for educational applications. For example, the act of scanning an AR code within a study guide or on a computer screen can offer supplementary visual explanations or animations that enhance the comprehension of the study material.

The utilization of AR codes can additionally foster enhanced social integration among individuals with hearing impairments. For instance, in the context of events or meetings, AR codes can display event schedules, welcome messages, or special announcements, facilitating improved navigation and fostering a greater sense of participation. Moreover, AR codes can be implemented in food service establishments, being positioned on menus or adjacent to individual dishes. By scanning these codes with a mobile device, users can access supplementary information regarding the dish, such as ingredient descriptions, cooking techniques, dish proportions relative to the table, and dietary restrictions [10].

This technology allows people with hearing impairments to make informed choices on their own, without relying solely on verbal communication or the help of others.

Potential Area for Future Research

Recognition of the common dignity of each individual is a central principle of contemporary social policy aimed at improving the lives of individuals with disabilities. Within this framework, there is a need not only to create mechanisms for social and labor inclusion, but also to ensure access to education and cultural resources for the development of this social segment. Thus, the successful integration of individuals with disabilities implies the development of specific cultural competencies.

Engaging persons with hearing impairments in cultural activities such as concerts, ballet or opera is an important aspect of promoting inclusiveness.

Currently, there is an increasing focus on haptic technology, which represents a promising direction for future developments in the field of devices and accessories geared towards individuals with hearing impairments. Haptic technology, referred to as tactile feedback technology, facilitates user-device interaction through touch. Its burgeoning popularity across industries such as gaming, healthcare, automotive, and virtual reality can be attributed to its ability to offer users immersive and interactive experiences [11].

Survey of the potential of haptic technology for future AR mobile app development

Haptic technology have been developed with the specific aim of enhancing the perceptual experience and compensating for the auditory information that individuals with hearing impairments may be unable to perceive. By providing tactile feedback, these accessories aim to enrich the sensory input and promote a more inclusive experience for individuals with hearing impairments.

Haptic clothing, although lesser-known, holds significant potential in addressing the needs of individuals with hearing impairments.

Vodafone, a telecommunications company, and Music is not Impossible, an innovation studio, have developed a vest accompanied by bracelets with the intent to enable individuals with partial hearing to attend music events. The haptic vest is equipped with 24 vibration sensors strategically positioned on the body, including bracelets on the wrists and ankles, to capture the rhythmic elements of the music directly from the stage. The battery-driven wireless wearable device facilitates the dissemination of vibrations within 8 discrete regions on the user's physique, thereby promoting what is referred to as an "Enveloping Body Perception." Every component receives intricate polyphonic auditory stimuli transmitted through the skin. Users are empowered to regulate the strength of the vibrations, which are visually depicted by means of adaptable color LED lights. What sets this invention apart is the utilization of Vodafone's high-speed 5G technology within the vest to collect sound information from the crowd. These data are then processed using advanced machine learning algorithms, transforming the collective noise into clear tactile responses [12].

Another example of vibration technology is – the global brand Desperados in cooperation with the wearable technology company CuteCircuit developed a tactile shirt called SoundShirt. The audio stream from the Desperados DJ console is processed by the CuteCircuit Q software, which converts music into various tactile sensations. Wearable sound shirts are equipped with built-in microactors that respond to music, generating forces of varying amplitude and character controlled by a DJ. The SoundShirt incorporates a total of 28 micro-actuators meticulously integrated into the fabric. These actuators are capable of wirelessly capturing and instantaneously receiving the music or sound in the form of data, as it is being performed live by the orchestra on stage. Consequently, the sensation of the trumpet's auditory impact traverses through the arms, while the vibrations generated by the drums resonate on the back. This comprehensive integration of auditory and haptic feedback enables a profoundly immersive experience for a deaf individual in the audience. The music is divided into different frequencies, including low, medium, medium, and high, each of which elicits distinct perceptual sensations [13].

Drawing on a comprehensive exploration of current advancements in the domain of haptic and vibration-based technology, along with sound-to-text conversion technologies, the authors of the article proposes to combine haptic technologies with augmented reality technology. Both technologies are compatible for integration into the mobile application. An application directly related to the smartphone’s camera and audio processing circuit can perform several functions:

1. transcribing sound information into both written text and sign language (depending on the user's preference) with the chosen translation displayed on the screen in real-time image mode;
2. converting music and ambient sounds into haptic sensations through different types of vibration;
3. supplementing the information with visual effects: illumination, dynamic color lighting and flashes, which correlate with the rhythm and mood of the music.

This application eliminates the need for specialized devices or accessories, thereby significantly simplifying its utilization and enabling individuals with hearing impairments to participate in various musical events without feeling the need to avoid them.

The aforementioned proposal represents the authors’ idea and recommendation. The proposed application can serve as a foundation for further research and the development of innovative solutions in the realm of augmented reality-integrated mobile applications catering to users with hearing impairments.

Conclusion. Therefore, the study demonstrated that augmented reality (AR) technologies such as smart glasses, mobile audio conversion applications into sign language, geospatial augmented reality, AR-codes and tactile accessories have the potential to facilitate information perception in individuals with hearing impairments. These technologies enable the creation of realistic visual and tactile cues that can substitute or complement auditory signals, providing users with equal access to sound content and communication.

The authors’ proposal to create a mobile application that combines some of the AR-technologies described above is an addition to existing technologies. Taking into account the fact that the smartphone is always within close reach of the user, a suitable and convenient option is to use mobile applications, which avoids the need to carry accessories or devices. This circumstance indeed confers unmistakable advantages, namely diminished physical exertion and augmented accessibility of functionalities. This application will afford individuals with hearing impairment the opportunity to not only visually and tactilely perceive musical performances through visualization and haptic feedback, but also to complement the visual effects. This approach presents novel prospects for advancing inclusive and equitable accessibility to music and artistic experiences for individuals with hearing impairments.

Thus, the use of AR-technologies and the creation of innovative mobile applications are promising approaches in facilitating the perception of information in persons with hearing impairments. These technologies have the potential to change the lives of many people by expanding their opportunities for communication, learning and entertainment. Further research and development in this field will contribute to the creation of a more accessible and inclusive society for all

References:

1. Smartphones – statistics & facts. URL: https://www.statista.com/topics/840/smartphones/#topicOverview
2. Statistics & facts – Augmented reality (AR). URL: https://www.statista.com/topics/3286/augmented-reality-ar/#topicOverview
3. Deafness and hearing loss. URL: https://www.who.int/news-room/fact-sheets/detail/deafness-and-hearing-loss
4. Ana Ćorić Samardžija. Mobile Augmented Reality Interactive Systems for Urban Tourism // Central European Conference on Information and Intelligent Systems. – 2015. pp. 129–130.
5. Ashley Miller, Joan Malasig, Brenda Castro, Vicki L. Hanson, Hugo Nicolau, Alessandra Brandão. The Use of Smart Glasses for Lecture Comprehension by Deaf and Hard of Hearing Students // CHI EA '17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems. – May 2017. pp. 1909–1915
6. Susanna Berra, Cláudia Pernencar, Flávio Almeida. Silent augmented narratives Inclusive Communication with Augmented Reality for deaf and hard of hearing // Media & Jurnalismo. – 2020. 20(36). pp. 174–185.
7. Michael Cabanillas-Carbonell, Piero Cusi-Ruiz, Daniela Prudencio-Galvez, José Luis Herrera Salazar. Mobile Application with Augmented Reality to Improve the Process of Learning Sign Language // International Journal of Interactive Mobile Technologies (iJIM). – 2020. 16(11). pp. 51–64.
8. Gerhard Schall. Mobile Augmented Reality for Human Scale Interaction with Geospatial Models // Diss. Graz. February 2011. pp. 50–55.
9. Becky Sue Parton, Robert Hancock, John Dawson. Augmented Reality for Deaf Students: Can Mobile Devices Make it Possible? // IFIP Human-Computer Interaction Symposium. HCIS 2010: Human-Computer Interaction pp. 309–312.
10. Shivam Sharma, Saud Shaikh. Augmented Reality in Human Life // International Journal for Research in Applied Science & Engineering Technology (IJRASET). – May 2022. Volume 10 Issue V. – pp. 3361–3362.
11. Haptic Technology Market Size to Surpass USD 17.49 Billion with Growing CAGR of 11.56% by 2032. URL: https://www.linkedin.com/pulse/haptic-technology-market-size-surpass-usd-1749-billion-rashmi-s
12. 5G haptic suits deliver unique live music experience for deaf and hard-of-hearing fans. URL: https://www.vodafone.com/news/technology/5g-haptic-suits-unique-live-music-experience
13. Feel Music on your Skin. URL: https://cutecircuit.com/soundshirt/