With the support of AI-based sign language recognition and sign linguistics theoretical framework, the beta version of SignTown released in May this year has accumulated over 8,500 users in Hong Kong and Japan. Launching SignTown has been the first key project of Project Shuwa which aims to break through the bottleneck of current sign language recognition models. The research team believes that the launch of SignTown is a great move in enhancing social inclusion as it will lay a solid foundation for further development of a sign language recognition model while, at the same time, raise public awareness towards sign language and promote social inclusion for the deaf community. 

SignTown is an online game in which players are placed in a fictional town where sign language is the official medium of communication. Players have to make signs in front of the camera to complete every task required in relation to daily activities, such as packing their bags for a trip, finding a hotel to stay in, or ordering food at a café. Meanwhile, the AI-powered recognition model will give immediate feedback on your signing accuracy. Cute hand-shaped characters scattered throughout the game will also explain to users the concepts of “sign language”, “deaf people” and “deaf culture”. In this way, both hearing and deaf people can learn the culture of signs and the deaf, and sign languages in Hong Kong and Japan in a fun and relaxing manner.

Automatic Sign Language Recognition and Project Shuwa 

Previous models of sign language recognition have not yielded a satisfactory accuracy rate because linguistic analysis of sign language has not yet been fully utilised in analysing the visual-gestural language data. In sign language, apart from hand movements, other gestural information such as body movements, facial expressions, head positions and movements, and mouth shapes play an equally important role in grammar. Exclusion of any of these parameters in a sign, a phrase, or a clause could result in ungrammatical or uninterpretable messages. While sign languages vary from one country to another, phonetic features like handshapes, orientations, and movements are universal and the number of possible combinations is finite and therefore recognition models are possible.  In order to recognise signing in a three-dimensional space with movement and depth, advanced equipment such as special 3D cameras and gloves with sensors are required, and as such it has been difficult to popularise this technology.

Project Shuwa has well acknowledged the above pain points and has successfully constructed the first machine-learning based model that can recognise three-dimensional sign language movements, and track and analyse hand and body movements as well as facial expressions using a standard camera. The new sign language game “SignTown” was built upon this novel technology. The next move in the project is to generate a sign dictionary that not only incorporates a search function, but also provides a virtual platform to facilitate sign language learning and documentation based on AI technology. The ultimate goal of the team is also to develop an automatic translation model that can recognise natural conversations in sign language and convert them into spoken language using the cameras of commonly used computers and smartphones.

Professor Felix SZE Yim Binh, Deputy Director of CSLDS, CUHK, said, “Our long-term goal is to create a barrier-free society that is truly inclusive for deaf signers or people who may benefit from signing communications. There is still a long way to go but ‘SignTown’ has opened the door to a lot more opportunities for people around the world to learn and know more about sign language in real life situations. There are plans to incorporate more signs in ‘SignTown’, so people around the world may learn more sign languages and know their differences. We are latching onto advanced technology to build an inclusive and harmonious society with equal opportunities.”  

Mr. Ichiro KABASAWA, Executive Director of The Nippon Foundation, said, “I would like to express my sincere appreciation to CUHK, Google, and Kwansei Gakuin University who came together and generously made SignTown a success. This would not have been possible without their involvement. I hope that this will lead to the realisation of an inclusive society, where deaf people, as members of society, can actively participate in social, economic, cultural, and other activities in sign language.”

Professor Katsuhisa MATSUOKA, Director of the Center for Sign Language Studies at Kwansei Gakuin University, said, “It is a great pleasure for our center to be involved in the technical development of Project Shuwa. This project is still in its infancy, so its potential is still unknown, but in the future, we may be able to develop machine translation between Japanese Sign Language and Hong Kong Sign Language facilitated by AI, and we may also be able to develop expert versions of learning tools for medical and legal professionals. These are all worth exploring.”

Representative of the technical exploration team from Google, Mr. Joe FRY said, “We are happy to have cooperated with The Nippon Foundation, CUHK and the Kwansei Gakuin University on such a meaningful project. Along with the growth of Project Shuwa, we envision more collaborative partnerships to join in accomplishing a greater goal of project development and technology advancement.”
 

For more information about the collaboration, please visit websites below:

Project Shuwa: http://projectshuwa.org

SignTown: https://sign.town

Source code for sign language recognition technology used in SignTown: https://github.com/google/shuwa

About the International Day of Sign Language 

23 September is the “International Day of Sign Languages”. It was adopted by the United Nations General Assembly on 19 December 2017, as a day to call for the recognition and importance of sign language, the language of the deaf. 23 September is the day the World Federation of the Deaf (WFD) which was founded in 1951. WFD is the world’s oldest international-level disability organisation dedicated to promoting the participation of deaf people in society through sign language. Currently, its affiliated organisations for deaf people can be found in more than 120 countries around the world.

About the Centre for Sign Linguistics and Deaf Studies of CUHK 

The Centre for Sign Linguistics and Deaf Studies was established at CUHK in November 2003, and is one of the research centres under the Department of Linguistics and Modern Languages. Since its establishment, CSLDS has been running a wide range of research projects and training programmes to promote sign linguistics as an academic discipline in Hong Kong and other parts of Asia. It also aims at promoting the use of sign language in deaf education, empowering deaf signers through academic training, improving the hearing-deaf communication by popularising sign language learning at the community level, and advising local governments/authorities on policy reforms that benefit the deaf communities. 

About The Nippon Foundation 

Since its establishment in 1962, the Nippon Foundation has been Japan’s largest social-contribution foundation, supporting public-interest projects that transcend both domains and national borders. In order to realise “a society where everyone supports everyone”, the Nippon Foundation is active in domains including those concerned with children, disabilities, disasters, oceans, and international cooperation. https://www.nippon-foundation.or.jp/.

About Kwansei Gakuin University 

Kwansei Gakuin University, founded in 1889, is a comprehensive university which provides education based on the principles of Christianity. The Sign Language Research Center (SLRC) was established in April 2015 as the first institution of sign language at a Japanese university, and it has been active since 2016, supported by The Nippon Foundation. The SLRC has been contributing to this project by selecting Japanese Sign Language (JSL) words and collecting JSL data for use in machine learning.