By Associated Press Published: 00:54 EDT, 7 September 2018 | Updated: 04:44 EDT, 7 September 2018 TROY, N.Y. (AP) - To learn Chinese in this room, talk to the floating panda head. The Mandarin-speaking avatar zips around a 360-degree restaurant scene in an artificial intelligence-driven instruction program that looks like a giant video game. Rensselaer Polytechnic Institute students testing the technology move inside the 12-foot-high, wrap-around projection to order virtual bean curd from the panda waiter, chat with Beijing market sellers and practice tai chi by mirroring moves of a watchful mentor. "Definitely less anxiety than messing it up with a real human being," says Rahul Divekar, a computer science graduate student working on the project. "So compared to that anxiety, this is a lot more easy." In this Aug. 22, 2018 image taken from video, Rensselaer Polytechnic Institute graduate student Xiangyang Mou practices tai chi with an avatar in a campus studio at Rensselaer Polytechnic Institute in Troy, N.Y. The "Mandarin Project" is a joint venture of RPI and IBM. Cognitive and Immersive Systems Laboratory researchers are developing a sort of smart room that can understand students' words, answer their questions and perceive their gestures. (AP Photo/Michael Hill) The "Mandarin Project" is a joint venture of RPI and IBM. Cognitive and Immersive Systems Laboratory researchers are developing a sort of smart room that can understand students' words, answer their questions and perceive their gestures. Lessons are presented as games or tasks, like ordering a meal out. Divekar orders Peking duck - "Beijng kaoya" - and the panda fetches the virtual dish. Divekar says the food was good - "Cai hen hao chi" - but he can't pay the bill. No problem, the panda replies - "ni keyi xi pan zi" - you can wash the dishes. Other scenes include an outdoor market and a garden, each a high-tech twist on cultural immersion. "Our plan is to complete several scenes of real life in China, to let the student be able to have a virtual trip over there," says Hui Su, director of the lab at RPI. Tests on the room with students studying Mandarin will continue this school year as they work on additional scenes, including an airport. A six-week course is being readied for the summer. Artificial intelligence, or AI, is ubiquitous in everything from call-center chat-bots to home assistants. Even some language instruction products on the market feature AI or virtual reality. The Mandarin Project is notable for its scale and sophistication. Computers simultaneously interpret speech and gesture to keep a dialogue going. When a student points to a picture and asks "What's that?" computers can come up with an answer. And feedback is immediate. When Divekar orders "dou fu" -or tofu - a voice responds "here's how close you got" and illustrates it with a graph of his intonation. Another voice gives the precise pronunciation. Still, language teachers need not fear for their jobs just yet. Developers of the Mandarin Project say it isn't sophisticated enough right now to completely replace classroom instruction. RPI president Shirley Ann Jackson foresees the same type of technology being applied to other spaces, such as corporate boardrooms. When the executives discuss a potential acquisition, the room will follow the group discussion and produce relevant information seamlessly into the debate. "We're not at the end of the line," Jackson says, "but closer to the beginning." In this Aug. 22, 2018 image taken from video, Rensselaer Polytechnic Institute graduate student Rahul Divekar interacts with a panda avatar that helps teach students Mandarin at Rensselaer Polytechnic Institute in Troy, N.Y. The "Mandarin Project" is a joint venture of RPI and IBM. Cognitive and Immersive Systems Laboratory researchers are developing a sort of smart room that can understand students' words, answer their questions and perceive their gestures. (AP Photo/Michael Hill) In this Aug. 22, 2018 image taken from video, Rensselaer Polytechnic Institute graduate students Xiangyang Mou, left, and Rahul Divekar interact with a panda avatar that helps teach students Mandarin, at Rensselaer Polytechnic Institute in Troy, N.Y. The "Mandarin Project" is a joint venture of RPI and IBM. Cognitive and Immersive Systems Laboratory researchers are developing a sort of smart room that can understand students' words, answer their questions and perceive their gestures. (AP Photo/Michael Hill) Advertisement
Rensselaer Polytechnic Institute and IBM have partnered on an AI - powered initiative that helps students learn Chinese in an immersive , virtual environment . The Mandarin technology enables students to interact with strangers in smart rooms that recreate real - world environments in China , such as restaurants or hotels . The technology can understand strangers ’ words , answer questions and interpret questions , which help keep the dialogue flowing .