The text in the video above reads as follows: Welcome to SignTown! An interactive experience where you can learn sign language with a little help from AI. Like how to order at a restaurant ('milk tea?'). Or checking into a hotel and requesting shampoo or soap. How does it work? All it takes is a webcam and machine learning to detect your body poses, facial expressions and hand movements. Give it a try now at www.sign.town
Google has spent over twenty years helping to make information accessible and useful in more than 150 languages. And our work is definitely not done, because the internet changes so quickly. About 15% of searches we see are entirely new every day. And when it comes to other types of information beyond words, in many ways, technology hasn’t even begun to scratch the surface of what’s possible. Take one example: sign language.
The task is daunting. There are as many sign languages as there are spoken languages around the world. That’s why, when we began exploring how we could better support sign language, we started small by researching and experimenting with what machine learning models could recognize. We spoke with members of the Deaf community, as well as linguistic experts, working closely with our partners at The Nippon Foundation, The Chinese University of Hong Kong and Kwansei Gakuin University. We began combining several ML models to recognize sign language as a sum of its parts — going beyond just hands to include body gestures and facial expressions.
After 14 months of testing with a database of videos for Japanese Sign Language and Hong Kong Sign Language, we launched SignTown: an interactive desktop application that works with a web browser and camera.
SignTown is an interactive web game built to help people to learn about sign language and Deaf culture. It uses machine learning to detect the user's ability to perform signs learned from the game.
SignTown is only one component of a broader effort to push the boundaries of technology for sign language and Deaf culture, named “Project Shuwa” after the Japanese word for sign language (“手話”). Future areas of development we’re exploring include building a more comprehensive dictionary across more sign and written languages, as well as collaborating with the Google Search team on surfacing these results to improve search quality for sign languages.
Advances in AI and ML now allow us to reliably detect hands, body poses and facial expressions using any camera inside a laptop or mobile phone. SignTown uses the MediaPipe Holistic model to identify keypoints from raw video frames, which we then feed into a classifier model to determine which sign is the closest match. This all runs inside of the user's browser, powered by Tensorflow.js.
We open-sourced the core models and tools for developers and researchers to build their own custom models at Google IO 2021. That means anyone who wants to train and deploy their own sign language model has the ability to do so.
At Google, we strive to help build a more accessible world for people with disabilities through technology. Our progress depends on collaborating with the right partners and developers to shape experiments that may one day become stand-alone tools. But it’s equally important that we raise awareness in the wider community to foster diversity and inclusivity. We hope our work in this area with SignTown gets us a little closer to that goal.
Maandishi katika video hapo juu yasomeka kama ifuatavyo: Karibu kwenye SignTown! Mazoezi ya kuingiliana ambayo unaweza kujifunza lugha ya ishara na msaada mdogo kutoka kwa AI Kama jinsi ya kuagiza katika mgahawa ('maziwa chai?'). Au ukitembelea hoteli na kuomba shampoo au sabuni. Inafanya kazije? Inahitaji tu kamera ya wavuti na kujifunza mashine kugundua mitazamo ya mwili wako, usemi wa uso na harakati za mkono. Google imekuwa ikijitahidi kwa miaka zaidi ya ishirini kusaidia kufanya habari iweze kupatikana na kuwa na manufaa katika lugha zaidi ya 150. Kazi yetu haijakamilika kwa sababu mtandao unabadilika haraka. Karibu 15% ya utafutaji unaofanywa kila siku ni mpya kabisa. Na linapokuja aina nyingine za habari zaidi ya maneno, kwa njia nyingi, teknolojia haijaanza hata kukwaruza uso wa kile kinachowezekana. Kwa mfano, fikiria lugha ya ishara. Kazi hiyo ni ngumu sana. Kuna lugha nyingi za ishara kama vile kuna lugha zinazozungumzwa ulimwenguni pote. Ndiyo sababu, wakati sisi kuanza kuchunguza jinsi tunaweza bora kusaidia lugha ya ishara, sisi kuanza ndogo kwa utafiti na majaribio na nini mashine kujifunza mifano inaweza kutambua. Kwa mujibu wa ripoti hiyo, wataalamu wa lugha na lugha wa Marekani wamekuwa wakifanya kazi kwa karibu na shirika la Nippon Foundation, Chuo Kikuu cha China cha Hong Kong na Chuo Kikuu cha Kwansei Gakuin. Tulianza kuchanganya mifano kadhaa ya ML kutambua lugha ya ishara kama jumla ya sehemu zake - kwenda zaidi ya mikono tu kujumuisha ishara za mwili na usemi wa uso. Baada ya miezi 14 ya majaribio na hifadhidata ya video kwa lugha ya ishara ya Kijapani na lugha ya ishara ya Hong Kong, tulizindua SignTown: programu ya desktop ya maingiliano ambayo inafanya kazi na kivinjari cha wavuti na kamera. SignTown ni mchezo wa mtandao wa maingiliano uliojengwa ili kuwasaidia watu kujifunza kuhusu lugha ya ishara na utamaduni wa viziwi. Inatumia kujifunza mashine kugundua uwezo wa mtumiaji wa kufanya ishara zilizojifunza kutoka kwa mchezo. SignTown ni sehemu moja tu ya juhudi kubwa ya kusukuma mipaka ya teknolojia kwa lugha ya ishara na utamaduni wa viziwi, iliyoitwa <unk>Project Shuwa<unk> baada ya neno la Kijapani kwa lugha ya ishara (<unk>手話<unk>). Maendeleo ya baadaye tunayochunguza ni pamoja na kujenga kamusi kamili zaidi katika lugha zaidi za ishara na zilizoandikwa, na pia kushirikiana na timu ya Utafutaji wa Google juu ya matokeo haya ili kuboresha ubora wa utafutaji wa lugha za ishara. Maendeleo katika AI na ML sasa yanaturuhusu kutambua mikono, nafasi za mwili na usemi wa uso kwa kutumia kamera yoyote ndani ya kompyuta ndogo au simu ya mkononi. SignTown hutumia mfano wa MediaPipe Holistic kutambua pointi muhimu kutoka kwa fremu za video mbichi, ambazo kisha tunaleta katika mfano wa uainishaji ili kuamua ni ishara gani inayofanana zaidi. Yote hii anaendesha ndani ya kivinjari cha mtumiaji, powered na Tensorflow.js. "Kama chanzo wazi cha mifano ya msingi na zana kwa watengenezaji na watafiti kujenga mifano yao wenyewe ya kawaida katika Google IO 2021.""" Hii inamaanisha kwamba mtu yeyote anayetaka kuunda na kutumia lugha ya ishara ya kibinafsi anaweza kufanya hivyo. """Tunatafuta kusaidia kujenga ulimwengu unaopatikana zaidi kwa watu wenye ulemavu kupitia teknolojia." Maendeleo yetu yanategemea kushirikiana na washirika sahihi na watengenezaji kuunda majaribio ambayo siku moja yanaweza kuwa zana za kujitegemea. Lakini ni muhimu pia kwamba tuongeze ufahamu katika jamii pana ili kukuza utofauti na ujumuishaji. Tunatumaini kazi yetu katika eneo hili na SignTown inatufanya tuwe karibu kidogo na lengo hilo.