Design and implementation of sign language translation program using motion recognition

About 5% of world population, 466million people in figure (432million adults and 34 million children), have the hearing impairment. Most people with hearing impairment use sign language as a means to communicate not only with people without disability but also with people with disability. However, there are hundreds of different sign languages and most people without disability do not know how to use sign language at all. Thus, it is difficult for people without disability to communicate with people with hearing impairment in daily living. This study aims to help those with hearing impairment for easier communication. To do this, this study designed a sign language translation program and implemented some of the design to recognize the user's motion by using motion recognition sensors and translating the recognized motions after finding the right words for the motions followed by displaying the words on the screen.

body movements to complete the meaning to express [3]. Although communication between people with and without hearing impairment is needed, it is difficult for people without hearing impairment to learn sign language and communicate with people with hearing impairment. Thus, it is necessary for people with hearing impairment to have a system that translates sign language into general texts or voice to solve the problem and communicate with people without hearing impairment [6] [7].
Although those who have inconvenient communication or pain due to physical disability may improve their quality of lives using assisted equipment utilizing information communication technology (ICT) such as Internet of Things (IoT), virtual reality (VR), augmented reality (AR) and artificial intelligence (AI), most of those technologies are focusing on people without disability [8].
Thus, this study designed a sign language translation program using motion recognition, and implemented the program. The sign language translation program implemented in this study recognizes sign language using a leap motion controller that identifies hand motions, and the results of the recognized sign language are checked via a monitor. Thus, the sign language translation program using motion recognition implement in this study is used efficiently when sign language interpretation is needed for family members or friends of people with hearing impairment. Furthermore, it can be used to learn sign language using motion recognition. Figure. 1 shows the significant results among the survey results, which aimed to determine the validity of the development in this study.

Figure 1. Survey
The survey 1 results tell us that only 7.4% of them knew persons with hearing impairment around them, which were not too many. Also, the majority (68.5%) of respondents communicated with persons with hearing impairment by "writing notes", which was convenient for the general public, whereas only a few respondents used sign language or body gestures. This survey result indicated that more people (68.5%) felt the need to learn sign language than those who did not. As shown in the above survey results, only 13% of the respondents said they understood and used sign language to communicate with persons with hearing impairment, which was proportionally very small. This means other assisted means is needed for the general public to communicate with people with hearing impairment. The survey told us that 7.4% of the respondents knew people with hearing impairment around them and among them, 29% had communication with people with hearing impairment, which was not too high. Nonetheless, 68.5% for the respondents recognized the need of sign language. However, it is really not viable for around 70% of people without hearing impairment to learn and use sign languages. Thus, some tools to assist communication with people with hearing impairment are needed, but writing notes is not the answer due to its limitation in speed and expressions. If some sort of translating machine is available as an intermediate medium to output texts or voice in real time while people with hearing impairment are doing sign language, it would make conversation faster and increase the depth and diversity of conversation as more complex expression than simple words can be possible.

Benchmarking
This study analyzed some products of currently available sign language interpretation for the design of the proposed program. Fig. 2 shows the sign language interpretation products that are benchmarked in this study. Parts used in a smart glove developed by engineers at the University of California San Diego are easily available in the market and assembled without difficulty [9]. Its price is less than US $100 since it uses inexpensive parts and can be utilized in other wearable products. It does not need a camera to deliver information through Bluetooth. The "sign language recognition AI secretary", which is a sign language interpretation product, is a system manufactured utilizing Google's TensorFlow AI open program [10]. It translates sign language into texts by Google's text-voice conversion software when users do sign language, and texts are converted into voice. "Motion Savvy" is a sign language interpreter based on leap motion sensor and Windows operating system-based tablet [11]. It recognizes sign language by three-dimensional (3D) sensor and converts it to text or voice. Motion Savvy can recognize the movement of hands and fingers within a 30 cm diameter with up to 0.01 mm precision. It is embedded with voice recognition software called "Nuance Engine" and can be used anywhere if the tablet is available. However, it can recognize and translate sign language only when the counterpart of the conversation is located at the same place.
The common characteristics of the above sign language interpreters benchmarked in this study are they required devices to recognize hand gesture in sign language and relatively high development cost. As various human-computer interaction (HCI) sensors emerge and AI technology advances, studies on sign language recognition have been diversely conducted.
This study aimed to use an assisted device that can be employed by users as many as possible and a rate of motion recognition was high during sign language translation. It designed a program using leap motion, which was relatively inexpensive, by referring to the benchmarking results. In addition, this study designed and implemented the program using leap motion because leap motion was considered as a suitable assisted device as it had a high recognition rate of positions and even precise motions of hands and fingers. Fig. 3 shows the flow chart of the "sign language translation program" implemented in this study.

Figure 1. Flow Chart of Translation Program
When the "sign language translation program" is run, its main screen is first displayed. Then, a user may click one of the contents: "Sign Language Translation", "Chatting", and "Training Video"from the main screen to use the program. When "Sign Language Translation" is selected, the program recognizes the user's motion in the real-time by the sensor in the linked program. If the motions are recognized as sign language, the motions are translated into words that were matched with the motions. Then, the translated words are entered into the chatting log and displayed in the user and other user's screens. The above process is iterated until it is canceled, and if the motions are not recognized as sign language, motion recognition continues iteratively to recognize the motions. When "Chatting" is selected, an input window pops up so that a user without disability may input a message to be sent to a person with hearing impairment. Once the send button is clicked, the message is entered into the chatting log and displayed. When "Training Video" is selected, a sign language training video content is provided by which people without disability can learn, and a user can select his/her preferred sign language word to learn.    4 shows the UI design of the main page, which is the first screen when a user is connected to the program. It was designed with simple functions as much as possible, and only translation, chatting, and training video functions are available. When a button is clicked, it connects to a page according to the selected button. Fig. 4 shows the UI of the sign language translation page. The upper part shows the screen that recognizes hand gestures using leap motion, and texts that translate sign language are displayed in the translation screen.  Basically, the "Sign Language Translation Program" designed in this study is operated by two users. A user (with hearing impairment) and other user (without hearing impairment) can exchange data through mutual communication in the same space. The conversation between users is inputted to the chatting log and both of the users can see all the conversation details. Users may use three features in the program: translation, chatting, and training video. The translation and chatting functions are connected to a conversation database while training video is connected to a video database.

Implementation
This study implemented only sign language translation function using leap motion without using the main page and smartphones or tablet computers, which were and assisted devices. The sign language translation algorithm in this study was implemented to recognize sign language through changes in coordinates of finger joints. Fig. 6 shows the screen that displays the first start coordinate of fingers and coordinates, which are changing in real-time. Null value is displayed since hands are not yet recognized after program execution.

Figure 4. Implementation Screen
The first coordinate is entered to the start coordinate array when each of the fingers is recognized, and a coordinate of each finger moved in real-time is entered into the moving coordinate array to display it on the screen continuously as shown in Fig. 7. A position of finger is displayed on the screen as a small circle using the Ellipse function. Only X and Y coordinates can be differentiated using a circle displayed on the screen.  Fig. 8 shows a screen of the translated word estimated using the motion recognition by comparing the start and current coordinates of each finger. The word the translated on the screen is "forward", and the message "forward" is displayed on the lower center on the screen when a difference in Z coordinate between start and current coordinates of each finger is over 500.

Conclusion
A sign language translation program utilizing motion recognition was developed in this study. The program developed in this study was implemented using leap motion without using other assisted devices. The number of people with hearing impairment is increasing according to recent statistics, and sign language interpreters who belong to sign language interpretation centers of local governments are providing interpretation services. However, the satisfaction level of sign language interpretation provided by the sign language interpretation centers felt by people with hearing impairment is not too high. Thus, a personalized portable sign language interpretation machine needs over ever. The authors of this study thought that it would be an effective sign language interpretation machine if the sign language interpretation function is embedded in portable smart devices.
The program implemented in this study displayed the translated word "forward" as a text by recognizing the user motion by the sensor if a certain distance or length of finger movement is recognized by comparing the change in Z coordinate value between the first recognized finger and later changing finger coordinates. For future research, this study aims to implement more complex sign language translation by adding a comparison of X and Y coordinates to enable more detailed measurement. The current implementation also recognized only one hand, but future study aims to recognize both hands thereby increasing the completeness of the design and implementation of the program by enabling a wide range of sign language correctly further.