AI-powered learning bridging the Deaf and hearing worlds
SignTown is the world’s first multi-language online sign language game.
CUHK, in collaboration with Google and Japanese counterparts, has launched the world’s first multi-language online sign language game. SignTown allows players to learn signs in an engaging online environment by harnessing machine-learning that recognises three-dimensional sign language movements, heralding a significant step towards promoting social inclusion for the Deaf community.
Nearly one in every thousand people worldwide is born with hearing loss or experience it later in life. More than 90% of these are born to hearing parents, creating an urgent need for hearing people to learn sign language in order to communicate with Deaf loved ones. However, hearing people often find it difficult to master the grammatical nuances of sign language, including hand and finger coordination, facial expressions, head positions, and mouth shapes. Get one element wrong, and the message often will be misunderstood.
Stimulating learning through an immersive and enjoyable environment
Watch the video to learn more about SignTown.
SignTown is an initiative of Project Shuwa, an international cross-disciplinary collaboration between CUHK, Google, The Nippon Foundation and Kwansei Gakuin University that aims to bridge the gap between the Deaf and hearing worlds via advanced technologies.
The innovative game places players in a fictional town where sign language is the official medium of communication. Players must make sign language movements in front of their laptop camera to complete day-to-day tasks, such as packing for a trip, finding a hotel, or ordering food. On-screen actors help players learn and imitate the relevant signs, and the AI-powered recognition model gives immediate feedback on the player’s accuracy.
SignTown significantly improves on existing sign language recognition models, which typically require advanced equipment such as special 3D cameras and gloves with sensors to capture sign movements in three-dimensional space. It harnesses the power of machine-learning to recognise three-dimensional sign language gestures, and track and analyse hand and body movements as well as facial expressions, using just a laptop camera.
The game can be used to learn both Hong Kong and Japanese sign languages, and also educates players about Deaf people and their culture. The official version of SignTown released in September 2021 has attracted over 12,000 users in Hong Kong and Japan.
Removing the barriers between Deaf and hearing people
Prof. Felix Sze, Deputy Director of the Centre for Sign Linguistics and Deaf Studies at CUHK, said, ‘Our long-term goal is to create a barrier-free society that is truly inclusive for Deaf signers or people who may benefit from signing communications. There is still a long way to go but SignTown has opened the door to a lot more opportunities for people around the world to learn and know more about sign language in real life situations. We are latching onto advanced technology to build an inclusive and harmonious society with equal opportunities.’
Prof. Felix Sze (left) hopes that Project Shuwa will promote social inclusion for the Deaf community.
Project Shuwa’s next ambition is to generate an online sign dictionary that does not only incorporate a search function, but also provides a virtual platform to facilitate sign language learning and documentation based on AI technology. The team’s ultimate aim is to develop an automatic translation model, similar to Google Translate, that can recognise natural conversations in sign language via smartphone cameras and convert them into speech.
Read more: