I recently watched a fascinating Horizon episode titled “Is Seeing Believing” (aired October 18, 2010). It was an exploration of the weird world of illusions and how our senses can play tricks on us.
One of the cool things demonstrated in the programme is something known as the McGurk effect. This is a perceptual phenomenon in which hearing and vision are linked when a person is recognising speech.
Watch the video (starts at about 31 seconds). You will hear a word being repeated. After a while, just due to the changes to the shapes made by the person’s mouth, you’ll hear a different word. All this without the audio actually changing!
Interestingly, by studying the McGurk effect, more accurate speech recognition computer programs have been created that use a video camera with lip reading software.