Using Sound to Help Blind People See

In the video below, Professor Shinsuke Shimojo from Caltech introduces this astounding device:


“Auditory regions are activated upon hearing sound, as are the visual regions, which we think will process the sound for its spatial qualities and elements. The visual part of the brain, when processing images, maps objects to spatial location, fitting them together like a puzzle piece,” Stiles says.

To learn more about how the crossmodal processing happens in the brain, the group is currently using functional magnetic resonance imaging (fMRI) data to analyze the crossmodal neural network.

These preexisting neural connections provide an important starting point for training visually impaired people to use devices that will help them see. A sighted person simply has to open their eyes, and the brain automatically processes images and information for seamless interaction with the environment.

Current devices for the blind and visually impaired are not so automatic or intuitive to use, generally requiring a user’s full concentration and attention to interpret information about the environment. The Shimojo lab’s new finding on the role of multimodal processing and crossmodal mappings starts to address this issue.


Beyond its practical implications, Shimojo says, the research raises an important philosophical question: What is seeing?

“It seems like such an obvious question, but it gets complicated,” says Shimojo. “Is seeing what happens when you open your eyes? No, because opening your eyes is not enough if the retina [the light-sensitive layer of tissue in the eye] is damaged.

“Is it when your visual cortex is activated? But our research has shown that the visual cortex can be activated by sound, indicating that we don’t really need our eyes to see. It’s very profound—we’re trying to give blind people a visual experience through other senses.”

The National Science Foundation, the Della Martin Fund for Discoveries in Mental Illness, and the Japan Science and Technology Agency, Core Research for Evolutional Science and Technology funded the work.


Source: Reproduced in full from under the Creative Commons Attribution 4.0 International license with a new title, additional article links removed, and page transition sentences added. Originally posted on by 

Featured Image Credit: Cheo70/Flickr