Saving Mum's Voice
AI Speech for the Vocally Stricken
On 19 October 2022, Prof. Lee Tan, Professor at the Department of Electronic Engineering and the Associate Dean (Education) of the Faculty of Engineering, CUHK, and the Dean of Students of S.H. Ho College, spoke to 240 attendees, including College members, students, alumni and staff members, on “Saving Mum’s Voice – AI Speech for the Vocally Stricken” in Ho Sin Hang Hall during the College’s second High Table Dinner.
Prof. Lee started the talk with the story of Jody, a 60-year-old lady who had her tongue and vocal cords totally removed last year because of a recurring tongue cancer. The surgery meant that she would lose her voice forever. Before the surgery took place, the girlfriend of Jody’s son sought advice on an online forum about how to clone Jody’s voice by artificial intelligence. The post caught the attention of Mr. Matthew Ma, a doctoral student of the Department of Electronic Engineering of CUHK, who soon communicated with Prof. Lee on the possibility of helping Jody. A team at the department were developing the technology Jody’s family was looking for.
A video conference with Jody was quickly arranged. She was invited to do voice recording for around 5 hours in the laboratory. Some texts were prepared for Jody to read which included conversations that the team had developed before. There were also stories for children, as well as verses from the Bible which were requested by Jody. In order to rebuild a patient’s voice, the texts must consist of all the important elements of Cantonese speech, like initials, vowels, tones, word chunks, stresses, and Jody’s speaking style. With all these data collected, the team started to train the speech synthesis model. After a month’s trial and error, the team successfully built up samples and model that sounded highly similar to Jody’s voice. Prof. Lee also invited a collaborator to design a mobile application so that Jody could input texts in her mobile devices and turn them into speeches anytime she wanted.
Artificial intelligence is a popular topic nowadays. Everyone knows that it creates productivity and will change our future. But are we aware of how artificial intelligence skills actually help individuals in real life? Prof. Lee brought about Jody’s story to pinpoint how artificial intelligence can mean a lot to some people. Big companies may not work on practical products that cater for a small group of people, like tongue cancer patients. It is even harder to find companies to develop products that serve Cantonese-speaking people. With this gap between the commercial considerations and the urgent needs in our lives, Prof. Lee highlighted that CUHK has a compelling obligation to step in to help patients. “We are engineers and just know signal processing and computer programme. We have no other way to cure the patients. The responsibility goes to medical doctors. But we can do a little thing. We want to keep her voice,” said Prof. Lee.
Cable TV filmed an episode on Jody’s story in its series Wish early this year. Jody later shared on social media that she wanted the technology to be widely known and benefit more people. Magical developments did happen in the subsequent months. Prof. Lee’s team received many enquires and requests from people and their families. Up to now, five patients, including a 70-year-old barrister and a piano teacher in her thirties, have joined their programme. Like Jody, both of them kept their unique voices by artificial intelligence.
Rebuilding voices for patients is meaningful but the team is facing a shortage of manpower. They cannot work full time on this. In this case, Prof. Lee is thinking of setting up a student service team in S.H. Ho College. Undergraduate students from different faculties can play vital roles in the project. Some work can be easily done, for example, helping the patients do the recording, transcribing the texts, or doing some basic programming tasks. At the end of the talk, Prof. Lee sincerely appealed to College students and invited them to serve patients in any possible way they can think of.
High table guests (from left) include Prof. Ian Morley, Dr. Lipika Alok Chatterjee, Prof. Philip Chiu, Master Mok, Prof. Lee Tan and Mr. Brain Chung.
Students pay attention to Prof. Lee’s talk.
A student follows up with Prof. Lee about the AI technology in the Q&A session.
Prof. Lee speaks to around 240 attendees. The talk is also simultaneously delivered to more students in Zoom Webinar.
