Have you ever wondered what it would sound like to listen through sixteen ears? This past March, hundreds of Central Illinois children and families experienced microphone-array augmented listening technology firsthand at the annual Engineering Open House (EOH) sponsored by the University of Illinois College of Engineering. At the event, which attracts thousands of elementary-, middle-, and high-school students and local community members, visitors learned about technologies for enhancing human and machine listening.
Listen up (or down): The technology of directional listening
Our team’s award-winning exhibit introduced visitors to several directional listening technologies, which enhance audio by isolating sounds that come from a certain direction. Directional listening is important when the sounds we want to hear are far away, or when there are many different sounds coming from different directions—like at a crowded open house! There are two ways to focus on sounds from one direction: we can physically block sounds from directions we don’t want, or we can use the mathematical tools of signal processing to cancel out those unwanted sounds. At our exhibit in Engineering Hall, visitors could try both.
This carefully designed mechanical listening device is definitely not an oil funnel from the local hardware store.
The oldest and most intuitive listening technology is the ear horn, pictured above. This horn literally funnels sound waves from the direction in which it is pointed. The effect is surprisingly strong, and there is a noticeable difference in the acoustics of the two horns we had on display. The shape of the horn affects both its directional pattern and its effect on different sound wavelengths, which humans perceive as pitch. The toy listening dish shown below operates on the same principle, but also includes an electronic amplifier. The funnels work much better for directional listening, but the spy gadget is the clear winner for style.
This toy listening dish is not very powerful, but it certainly looks cool!
These mechanical hearing aids rely on physical acoustics to isolate sound from one direction. To listen in a different direction, the user needs to physically turn them in that direction. Modern directional listening technology uses microphone arrays, which are groups of microphones spread apart from each other in space. We can use signal processing to compare and combine the signals recorded by the microphones to tell what direction a sound came from or to listen in a certain direction. We can change the direction using software, without physically moving the microphones. With sophisticated array signal processing, we can even listen in multiple directions at once, and can compensate for reflections and echoes in the room.
This post describes our new wearable microphone impulse response data set, which is available for download from the Illinois Data Bank and is the subject of a paper at ICASSP 2019.
Acoustic impulse responses were measured from 24 source angles to 80 points across the body.
Have you ever been at a crowded party and struggled to hear the person next to you? Crowded, noisy places are some of the most difficult listening environments, especially for people with hearing loss. Noisy rooms are also a challenge for electronic listening systems, like teleconferencing equipment and smart speakers that recognize users’ voices. That’s why many conference room microphones and smart speakers use as many as eight microphones instead of just one or two. These arrays of microphones, which are usually laid out in a regular pattern like a circle, let the device focus on sounds coming from one direction and block out other sounds. Arrays work like camera lenses: larger lenses can focus light more narrowly, and arrays with more microphones spread out over a larger area can better distinguish between sounds from different directions.
Wearable microphone arrays
Microphone arrays are also sometimes used in listening devices, including hearing aids and the emerging product category of smart headphones. These array-equipped devices can help users to tune out annoying sounds and focus on what they want to hear. Unfortunately, most hearing aids only have two microphones spaced a few millimeters apart, so they aren’t very good at focusing in one direction. What if hearing aids—or smart headphones, or augmented reality headsets—had a dozen microphones instead of just two? What if they had one hundred microphones spread all over the user’s body, attached to their clothing and accessories? In principle, a large wearable array could provide far better sound quality than listening devices today.
Over the years, there have been several papers about wearable arrays: vests, necklaces, eyeglasses, helmets. It’s also a popular idea on crowdfunding websites. But there have been no commercially successful wearable microphone array products. Although several engineers have built these arrays, no one has rigorously studied their design tradeoffs. How many microphones do we need? How far apart should they be? Does it matter what clothes the user is wearing? How much better are they than conventional listening devices? We developed a new data set to help researchers answer these questions and to explore the possibilities of wearable microphone arrays.
Augmented listening systems “remix” the sounds we perceive around us, making some louder and some quieter.
I am one of millions of people who suffer from hearing loss. For my entire life I’ve known the frustration of asking people to repeat themselves, struggling to communicate over the phone, and skipping social events because I know they’ll be too noisy. Hearing aids do help, but they don’t work well in the noisy, crowded situations where I need them the most. That’s why I decided to devote my PhD thesis to improving the performance of hearing aids in noisy environments.
As my research progressed, I realized that this problem is not limited to hearing aids, and that the technologies I am developing could also help people who don’t suffer from hearing loss. Over the last few years, there has been rapid growth in a product category that I call augmented listening (AL): technologies that enhance human listening abilities by modifying the sounds they hear in real time. Augmented listening devices include:
- traditional hearing aids, which are prescribed by a clinician to patients with hearing loss;
- low-cost personal sound amplification products (PSAPs), which are ostensibly for normal-hearing listeners;
- advanced headphones, sometimes called “hearables,” that incorporate listening enhancement as well as features like heart-rate sensing; and
- augmented- and mixed-reality headsets, which supplement real-world sound with extra information.
These product categories have been converging in recent years as hearing aids add new consumer-technology features like Bluetooth and headphone products promise to enhance real-world sounds. Recent regulatory changes that allow hearing aids to be sold over the counter will also help to shake up the market.
Je, umewahi kujiuliza ingekuwaje kusikiliza kupitia masikio kumi na sita? Mwezi Machi, mamia ya watoto na familia za Illinois ya Kati walipata uzoefu wa teknolojia ya kusikiliza iliyoongezwa na safu ya kipaza sauti katika Nyumba ya Uhandisi ya Mwaka iliyofadhiliwa na Chuo Kikuu cha Uhandisi cha Illinois. Katika tukio hilo, ambalo huvutia maelfu ya wanafunzi wa shule za msingi, za kati na za sekondari na wanachama wa jamii ya eneo hilo, wageni walijifunza juu ya teknolojia za kuboresha kusikiliza kwa binadamu na mashine. Kusikiliza juu (au chini): Teknolojia ya kusikiliza mwelekeo Maonyesho ya timu yetu ya kushinda tuzo iliwajulisha wageni teknolojia kadhaa za kusikiliza mwelekeo, ambazo zinaongeza sauti kwa kutenganisha sauti zinazokuja kutoka mwelekeo fulani. Kusikiliza kwa mwelekeo ni muhimu wakati sauti tunataka kusikia ni mbali, au wakati kuna sauti nyingi tofauti zinazokuja kutoka pande tofauti - kama katika nyumba iliyojaa wazi! Kuna njia mbili za kuzingatia sauti kutoka mwelekeo mmoja: tunaweza kimwili kuzuia sauti kutoka mwelekeo sisi si wanataka, au tunaweza kutumia zana za hisabati ya usindikaji ishara kufuta sauti hizo zisizohitajika. Katika maonyesho yetu katika Jumba la Uhandisi, wageni wangeweza kujaribu yote mawili. Kwa hakika kifaa hiki cha kusikiliza kilichobuniwa kwa uangalifu si mtungi wa mafuta kutoka kwenye duka la vifaa vya ujenzi la mahali hapo. Teknolojia ya zamani zaidi na yenye kueleweka zaidi ya kusikiliza ni pembe ya sikio, iliyoonyeshwa hapo juu. Pembe hiyo husafirisha mawimbi ya sauti kutoka upande unaoelekezwa. Athari ni yenye nguvu kwa kushangaza, na kuna tofauti inayoonekana katika sauti ya pembe mbili tulizokuwa nazo kwenye maonyesho. Umbo la pembe huathiri mwelekeo wake na athari zake kwa urefu tofauti wa mawimbi ya sauti, ambayo wanadamu huona kama sauti. Chombo cha kusikiliza cha kuchezea kilichoonyeshwa hapa chini hufanya kazi kwa kanuni ileile, lakini pia kina kipanukuu cha elektroniki. Funnels kazi bora zaidi kwa ajili ya kusikiliza mwelekeo, lakini spy gadget ni mshindi wazi kwa mtindo. Chombo hiki cha kuchezea cha kusikiliza si chenye nguvu sana, lakini kwa hakika kinaonekana kuwa kizuri! Vifaa hivyo vya kusikia hutumia vifaa vya sauti ili kutenganisha sauti kutoka upande mmoja. Ili kusikiliza katika mwelekeo tofauti, mtumiaji anahitaji kugeuza kimwili katika mwelekeo huo. Teknolojia ya kisasa ya kusikiliza kwa mwelekeo hutumia safu za maikrofoni, ambazo ni vikundi vya maikrofoni vilivyoenea mbali na kila mmoja katika nafasi. Kwa mfano, tunaweza kutumia vifaa vya kuchanganua ishara ili kulinganisha na kuchanganya ishara zilizorekodiwa na maikrofoni ili kujua sauti ilitoka upande gani au kusikiliza kwa mwelekeo fulani. Tunaweza kubadilisha mwelekeo kwa kutumia programu, bila kuhamisha maikrofoni kimwili. Kwa usindikaji wa ishara ya safu ya hali ya juu, tunaweza hata kusikiliza katika mwelekeo mbalimbali kwa wakati mmoja, na tunaweza kusawazisha kwa tafakari na echoes katika chumba. "Kama ilivyoelezwa hapo juu, ""Makampuni ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia ya teknolojia.""" Kwa mfano, majibu ya msukumo wa sauti yalipimwa kutoka pembe 24 za chanzo hadi pointi 80 kwenye mwili. Je, umewahi kuhudhuria karamu yenye watu wengi na kupata ugumu wa kumsikia mtu aliye karibu nawe? Maeneo yenye kelele na yenye watu wengi ni mahali penye matatizo ya kusikiliza, hasa kwa watu wenye matatizo ya kusikia. Vyumba vyenye kelele pia ni changamoto kwa mifumo ya kusikiliza ya elektroniki, kama vifaa vya mkutano wa simu na wasemaji wenye akili ambao hutambua sauti za watumiaji. Hiyo ndiyo sababu maikrofoni nyingi za chumba cha mikutano na wasemaji wenye akili hutumia maikrofoni nane badala ya moja au mbili tu. Mikrofoni hizo, ambazo kwa kawaida zimepangwa kwa mviringo, huruhusu kifaa hicho kukazia fikira sauti zinazotoka upande mmoja na kuzuia sauti nyingine. Lens kubwa inaweza kuelekeza mwanga kwa njia nyembamba zaidi, na lensi kubwa na mikrofoni nyingi zaidi zinazosambazwa katika eneo kubwa zinaweza kutofautisha sauti kutoka pande tofauti. Vipande vya kipaza sauti vinavyoweza kuvaliwa pia hutumiwa wakati mwingine katika vifaa vya kusikiliza, pamoja na vifaa vya kusikia na jamii ya bidhaa inayoibuka ya vichwa vya sauti vya akili. Vifaa hivyo vinaweza kuwasaidia watumiaji wasisikilize sauti zenye kusumbua na kukazia fikira kile wanachotaka kusikia. Kwa bahati mbaya, vifaa vingi vya kusikia vina maikrofoni mbili tu zilizogawanywa kwa milimita chache, kwa hiyo hazina uwezo wa kukazia fikira upande mmoja. Namna gani ikiwa vifaa vya kusikia - au vichwa vya sauti vya akili, au vichwa vya sauti vya hali halisi iliyoongezwa - vingekuwa na maikrofoni kumi na mbili badala ya mbili tu? Vipi ikiwa wangekuwa na maikrofoni mia moja zilizosambazwa kwenye mwili wote wa mtumiaji, zilizounganishwa na nguo na vifaa vyao? Kwa kanuni, safu kubwa inayoweza kuvaliwa ingeweza kutoa ubora bora zaidi wa sauti kuliko vifaa vya kusikiliza vya leo. Kwa miaka mingi, kumekuwa na karatasi kadhaa kuhusu safu za kuvaa: vikapu, mkufu, miwani, kofia za chuma. Pia ni wazo maarufu kwenye tovuti za crowdfunding. Lakini hakuna bidhaa zilizofanikiwa kibiashara za mikrofoni zinazoweza kuvaliwa. Ingawa wahandisi kadhaa wamejenga safu hizo, hakuna mtu aliyechunguza kwa uangalifu mapendekezo ya kubuni. Tunahitaji maikrofoni ngapi? Wanapaswa kuwa mbali kadiri gani? Je, ni muhimu kujua ni nguo gani ambazo mtumiaji amevaa? Je, ni bora kadiri gani kuliko vifaa vya kawaida vya kusikiliza? "Tumeunda seti mpya ya data kusaidia watafiti kujibu maswali haya na kuchunguza uwezekano wa safu za kipaza sauti zinazoweza kuvaliwa. """ Mifumo ya kusikiliza iliyoongezwa <unk> inachanganya <unk> sauti tunazotambua kuzunguka kwetu, ikifanya zingine kuwa kubwa na zingine kuwa za kimya. Mimi ni mmoja wa mamilioni ya watu wanaopoteza uwezo wao wa kusikia. Kwa maisha yangu yote nimejua kukata tamaa ya kuwaomba watu warudiane, kupambana kuwasiliana kwa simu, na kuruka matukio ya kijamii kwa sababu najua watakuwa na kelele sana. Vifaa vya kusikia husaidia, lakini havifanyi kazi vizuri katika hali zenye kelele, zenye umati ambapo ninazihitaji zaidi. Hiyo ndiyo sababu niliamua kujitolea thesis yangu ya PhD kuboresha utendaji wa vifaa vya kusikia katika mazingira yenye kelele. Kama utafiti wangu uliendelea, niligundua kwamba tatizo hili si tu kwa vifaa vya kusikia, na kwamba teknolojia mimi nina maendeleo inaweza pia kusaidia watu ambao hawana kuteseka na hasara ya kusikia. Katika miaka ya hivi karibuni, kuna ukuaji wa haraka katika jamii ya bidhaa ambazo mimi huita kusikiliza kuongezeka (AL): teknolojia ambazo zinaongeza uwezo wa kusikiliza wa binadamu kwa kurekebisha sauti wanazosikia kwa wakati halisi. Vifaa vya kusikiliza vilivyoongezwa ni pamoja na: - vifaa vya kawaida vya kusikia, ambavyo vimeagizwa na daktari kwa wagonjwa wenye upotezaji wa kusikia; - bidhaa za gharama nafuu za kuimarisha sauti (PSAPs), ambazo zinaonekana kwa wasikilizaji wa kusikia kawaida; - vichwa vya sauti vya hali ya juu, wakati mwingine vinaitwa "hearables" ambazo zinajumuisha kuboresha kusikiliza na pia sifa kama vile kuhisi kiwango cha moyo; na - vichwa vya sauti vya hali ya juu na mchanganyiko. Kwa miaka michache iliyopita, bidhaa hizi zimekuwa zikikubaliana, kwani vifaa vya kusikia vinaongeza huduma mpya za teknolojia ya watumiaji kama vile Bluetooth na bidhaa za vichwa vya sauti zinaahidi kuboresha sauti za ulimwengu halisi. Mabadiliko ya hivi karibuni ya kisheria ambayo huruhusu vifaa vya kusikia kuuzwa bila kuagiza pia yatasaidia kutikisa soko.