The Sound of Color

A completely colorblind musician and painter perceives the world in a new way with help from technology.

By Jef Akst | 

ORANGE TONES: Neil Harbisson wearing his “eyeborg,” a device that converts light waves into vibrations that lend a touch of color to his worldDan Wilton/RedBulletin

As a kid growing up in Barcelona, Spain, Neil Harbisson could tell you that the sky was blue, the grass was green, and a lemon was yellow. But he couldn’t tell you exactly what all those descriptions really meant. Born with a rare inherited condition  similar to the one that plagued the Pacific islanders neurologist Oliver Sacks wrote about in The Island of the Colorblind, Harbisson sees only in shades of gray, and had simply memorized the colors he thought he was supposed to know. But it wasn’t until he was 11 years old that he learned that he didn’t perceive the world in the same way as most people.

“I noticed that other students at school could identify colors easier than me,” he recalls. “Then I knew there was a problem with color.”

A decade later, as a music composition student at the Dartington College of Arts in England, Harbisson discovered there was hope that he might see things differently. In 2003, he attended a lecture about using technology to change the way we see the world. After the talk, Harbisson approached the speaker, young cybernetics innovator Adam Montandon, then at the University of Plymouth, to describe his condition and ask if there might be a way to help him perceive color.

On his train ride home, Montandon thought about the possibility of using a system that would assign different musical instruments to colors—drums for red and violins for blue, for example. But realizing that this would introduce his own interpretations of color, Montandon then considered a device that would simply say the names of colors aloud, but this didn’t sit right with him either. “I wanted to give him something a bit more magical,” Montandon recalls. Finally, he thought about the physical similarities of light and sound. “Light is a wavelength that moves very fast,” he says. “[If] you slow it down enough, it stops becoming visible. It starts becoming audible.”

In just 2 weeks’ time, Montandon and Harbisson created a device that translated the light waves that correspond to different colors into sounds with different pitches. The prototype, constructed from an inexpensive computer webcam, a laptop carried in a backpack, and a pair of old headphones, was a bit “crude,” Montandon admits. “It was fairly primitive, but it was good enough,” he says.

EYES AND EARS: The original version of the eyeborg used a computer webcam to detect light, a laptop to compute the conversion to sound, and headphones to transmit that sound to Harbisson’s ears.an Wilton/RedBulletin

Harbisson tested the device in a school hallway in front of a big red notice board as a friend pointed to different objects, identifying their colors. He quickly learned the color/pitch associations. “Then he just ran off down the corridor,” Montandon says. “I couldn’t stop him. He went to listen to absolutely everything.” And when Montandon returned to the school 2 weeks later, he learned that Harbisson “hadn’t even switched off the computer.”

Harbisson listened to the colors of the houses in the street. He went to local grocers’ shops to listen to the sounds of fruits and vegetables. “It’s like listening to electronic music,” says Harbisson, who now wears a refined version of the device, which he calls an “eyeborg.”

“And it’s constant,” he adds. “I wasn’t expecting that I would be listening to colors all the time, but there’s color absolutely everywhere.”

The device has since been through many iterations, with the help of Montandon; a software developer from Kranj, Slovenia, named Peter Kese; and most recently, Matias Lizana, a computer engineering student at the Universitat Politècnica de Catalunya in Barcelona. Today, the system comprises a camera that sticks out above Harbisson’s head like an antenna, and a small computer chip that converts light to sound. Harbisson has since also forgone the need for headphones, mounting the chip to the back of his head, where it transmits the sound vibrations directly into his skull bones. “I receive color through the bone, and I’m listening to you through the ears,” Harbisson says. This helps him “differentiate what is a visual sound and what is an audio sound.”

Harbisson can also hear many more colors than the handful portrayed by the original device. He can even hear colors of wavelengths not visible to the human eye, such as those in the infrared. This year, he plans to add ultraviolet. The project is the ultimate demonstration of the promise of cyborgs, says Montandon, now a digital technologies professor at Erhvervsakademiet Lillebælt in Denmark. “It’s very easy with the technology we have now to explore different parts of the spectrum that we don’t normally experience,” Montandon adds. “You’re not just enabling someone with a disability, but you’re enabling someone to be more than a regular human.”

Of course, the device is not the most practical medical solution to color blindness, its creators recognize. First, wearing a camera around all day is inconvenient. Furthermore, the device is really designed for those with true color blindness, or achromatopsia, which affects only about 1 in 30,000 people—not for those who simply are unable to distinguish red from green, a far more common condition. And people with achromatopsia “also have a whole host of other vision problems,” says Medical College of Wisconsin neuroscientist Joseph Carroll, including involuntary eye movements called nystagmus, very poor acuity, and photophobia. “The ability to perceive color is the least of their worries,” he says. A more promising treatment for these patients is gene therapy to restore cone function, and thus increase acuity and reduce photophobia, he adds. Human trials for these treatments are expected to start this year.

That said, Carroll adds, “this is really cool and adds a dimensionality to your visual experience.” Indeed, Harbisson’s eyeborg is catching the attention of some musicians and artists. The pianist Jools Holland, for example, used a version of the device to accompany live concerts held in 2009–2010 across the U.K., Montandon says. “He likes to improvise, [so] we created a reverse system that would turn his music into colors and lights as he played.”

And Harbisson himself, who now works full-time as a freelance “colorologist,” pairing colors with music, uses the device for artistic inspiration, often painting what he hears. Listening to his world has definitely changed how he perceives it, he says. “People said that cities were gray—they’re not. They are actually extremely colorful. I’m discovering color in a different way.”

Source: TheScientist
http://the-scientist.com/2012/05/01/the-sound-of-color/

Posted by
Robert Karl Stonjek