Researchers at the MIT Media Lab have created a finger-worn device with a built-in camera that can convert text to speech for the visually impaired. Researchers at the MIT Media Laboratory have built a prototype of a finger-mounted device with a built-in camera that converts written text into audio for visually impaired users. The device provides feedback—either tactile or audible—that guides the user's finger along a line of text, and the system generates the corresponding audio in real time. 
 
 "You really need to have a tight coupling between what the person hears and where the fingertip is," says Roy Shilkrot, an MIT graduate student in media arts and sciences and, together with Media Lab postdoc Jochen Huber, lead author on a new paper describing the device. "For visually impaired users, this is a translation. It's something that translates whatever the finger is 'seeing' to audio. They really need a fast, real-time feedback to maintain this connection. If it's broken, it breaks the illusion." 
 
 Huber will present the paper at the Association for Computing Machinery's Computer-Human Interface conference in April. His and Shilkrot's co-authors are Pattie Maes, the Alexander W. Dreyfoos Professor in Media Arts and Sciences at MIT; Suranga Nanayakkara, an assistant professor of engineering product development at the Singapore University of Technology and Design, who was a postdoc and later a visiting professor in Maes' lab; and Meng Ee Wong of Nanyang Technological University in Singapore. 
 
 The paper also reports the results of a usability study conducted with vision-impaired volunteers, in which the researchers tested several variations of their device. One included two haptic motors, one on top of the finger and the other beneath it. The vibration of the motors indicated whether the subject should raise or lower the tracking finger. 
 
 Another version, without the motors, instead used audio feedback: a musical tone that increased in volume if the user's finger began to drift away from the line of text. The researchers also tested the motors and musical tone in conjunction. There was no consensus among the subjects, however, on which types of feedback were most useful. So in ongoing work, the researchers are concentrating on audio feedback, since it allows for a smaller, lighter-weight sensor. 
 
 Researchers at the MIT Media Lab have created a finger-worn device with a built-in camera that can convert text to speech for the visually impaired. 
 
 Bottom line 
 
 The key to the system's real-time performance is an algorithm for processing the camera's video feed, which Shilkrot and his colleagues developed. Each time the user positions his or her finger at the start of a new line, the algorithm makes a host of guesses about the baseline of the letters. Since most lines of text include letters whose bottoms descend below the baseline, and because skewed orientations of the finger can cause the system to confuse nearby lines, those guesses will differ. But most of them tend to cluster together, and the algorithm selects the median value of the densest cluster. 
 
 That value, in turn, constrains the guesses that the system makes with each new frame of video, as the user's finger moves to the right, which reduces the algorithm's computational burden. 
 
 Given its estimate of the baseline of the text, the algorithm also tracks each individual word as it slides past the camera. When it recognizes that a word is positioned near the center of the camera's field of view—which reduces distortion—it crops just that word out of the image. The baseline estimate also allows the algorithm to realign the word, compensating for distortion caused by oddball camera angles, before passing it to open-source software that recognizes the characters and translates recognized words into synthesized speech. 
 
 In the work reported in the new paper, the algorithms were executed on a laptop connected to the finger-mounted devices. But in ongoing work, Marcel Polanco, a master's student in computer science and engineering, and Michael Chang, an undergraduate computer science major participating in the project through MIT's Undergraduate Research Opportunities Program, are developing a version of the software that runs on an Android phone, to make the system more portable. 
 
 The researchers have also discovered that their device may have broader applications than they'd initially realized. "Since we started working on that, it really became obvious to us that anyone who needs help with reading can benefit from this," Shilkrot says. "We got many emails and requests from organizations, but also just parents of children with dyslexia, for instance." 
 
 "It's a good idea to use the finger in place of eye motion, because fingers are, like the eye, capable of quickly moving with intention in x and y and can scan things quickly," says George Stetten, a physician and engineer with joint appointments at Carnegie Mellon's Robotics Institute and the University of Pittsburgh's Bioengineering Department, who is developing a finger-mounted device that gives visually impaired users information about distant objects. "I am very impressed with what they do." 
 
 Explore further: MIT finger device reads to the blind in real time 
 
 More information: The paper is available online: fluid.media.mit.edu/sites/defa … fonts%20embedded.pdf 
 
 ||||| At MIT’s Media Labs, researchers Roy Shilkrot, Jochen Huber and others are working on the “FingerReader,” a ring-like device that straps itself around your finger and reads printed text out loud with a synthesized voice, thanks to a mounted camera and heavily modified open source software. 
 
 The FingerReader’s voice is clipped and metallic – what one might liken to a Berlitz tape speaking through clenched teeth – but it is something the Finger Reader team is aware of and is working on, amongst many other things. The device itself is plasticine and upon first-hand inspection, weighs no more than a regular ring. To hear the words read aloud, simply point your finger at the text you wish to read. It’s possible thanks to a small camera mounted on the ring. 
 
 It doesn’t work with text as small as, say, on a medicine bottle or baseball box scores, but it can read 12-point printed text, the on-screen computer component shown in the video above highlighting each word to the sound of what Shilkrot described as “bings and beeps.” Stray too far from a line, and a dial tone-like noise increases. Remain on the line of text and you can hear it. And though – as a whole – the FingerReader sounds like a ‘Reading Pen,’ Huber and Shilkrot differentiate it from the Reading Pen, noting that it gives real time feedback, whereas the Reading Pen does not, and that their software can read a whole line of text, and not one word at a time, as is the case with a Reading Pen. 
 
 In an interview, Shilkrot, a Ph.D student, said he didn’t envision the final version of the ‘Finger Reader’ to be only for people with sight difficulties. He said it was for people with “disability, ability, and superability,” and suggested a scenario in which the Finger Reader translated text in another language. He said it was worth noting that the blind and visually impaired liked using devices that weren’t built just for them. He also pointed towards a study conducted by the Royal National Institute of the Blind in Britain that noted in 2011 that “only seven per cent of books are available in large print, unabridged audio and braille, including titles available in these formats as eBooks,” noting that an easy-to-use device would open up a lot of heretofore unread texts. 
 
 Though there is no immediate plan to take FingerReader to market, Shilkrot is “aware of the costs of devices these days. [Even so], we still can’t say how much the FingerReader will cost as a consumer product. The technology is still evolving around miniaturization, stand-alone wireless operation, tethering to a PC or smartphone.” 
 
 Video credit: FingerReader – Wearable Text-Reading Device from Fluid Interfaces on Vimeo.
– A new invention could help blind people "read" books and other written text without the need for Braille. MIT researchers have created a prototype of a finger-mounted device that uses a camera to scan text, which is then converted to speech in real time. "For visually impaired users, this is a translation," researcher Roy Shilkrot explains at Phys.org. "It's something that translates whatever the finger is 'seeing' to audio." A huge challenge, however, was figuring out how to keep visually impaired volunteers' fingers in line with text. Researchers came up with two options: a device that used vibrating motors on top and beneath the finger to indicate if it needed to move up or down, and another that issued musical tones as clues. While volunteers didn't favor one device over the other, researchers say the motorless option is smaller, lighter, and therefore more ideal. The device uses an algorithm to process what the camera perceives. For now, it relies on an attached laptop, but the next version may use a phone to make it more portable. In an interview last year with TechCrunch, Shilkrot said the invention should open up a huge number of books to the blind. And perhaps to those who aren't blind but just have trouble reading: Shilkrot says parents of kids with dyslexia have already reached out. An engineer is working on a similar device that could help blind people identify distant objects. (A blind man can see again thanks to a bionic eye.)