June 29, 2022

Japan’s local governments are planning to turn into artificial intelligence to improve communication with people who are deaf or hard of hearing at their counters for the public.

According to reports, the University of Electro-Communications in Tokyo and SoftBank Corp. have jointly developed a system that converts sign-language gestures into written text.

Although the system currently requires equipment at counters, the local municipalities, however, hope it will eventually be usable with just a smartphone.

About the AI system

The AI system dubbed “Sure Talk,” translates sign into Japanese text. It uses image recognition technology that analyzes the skeletal movements of several areas of the body, such as fingers and arms, to convert signs into Japanese.

To develop the system, it has used sign images of hundreds of people that were then digitized.

A Narashino city official commented that although conversation with people who are deaf or hard of hearing can be carried out in writing, the AI system is much smoother because the translation takes place in real-time.

The AI system still has a lot of room for improvement and currently, it can only accurately translate gestures into about 1,500 Japanese words.

According to a SoftBank engineer involved in developing the system, a huge amount of sign language data is needed to build a model for accurate translations of signs into Japanese text.

Hence, in order to improve the system’s accuracy, the mobile communications and internet services company has deemed it necessary to launch a website and smartphone app requesting cooperation from the public. It has also called upon as many people as possible to send in sign language images.

In other related news, Japan’s Hokkaido University and Nippon Telegraph and Telephone East Corp. are also jointly reported to be developing an AI-based automated sign-language translation system.

The Nippon Foundation and Google LLC have also reported to develop a game called “Sign Town,” to help people enjoy learning sign language.

In the game, players can advance a level when they can answer questions correctly using signs directed at computer cameras.

The Japanese Federation of the Deaf however argued that for AI to be an effective tool for people who are deaf and hard of hearing, a lot more nuance is necessary, something which will be difficult to achieve in the short term if at all.