ProjectsEducation

Decoding Sound

Multi-city workshops on how we hear, and how to tell a computer.

Sound is the energy, the precondition, which first enabled language: a code.

In these sessions we explore and decode the fundamentals of sound, understand the uses of filters and effects, and confront the limits of our perception through auditory illusions in psychoacoustics.

It’s a chance to see sound in a new light and explore how technology can help machines recognise acoustic information in space and time. The goal is to develop the fundamental semantics of sound, and how technology can use them to make things happen.

The series ran in several cities, focused on the language and syntax of human sound perception and ways to communicate it to a computer using the visual programming language Max/MSP.

The workshop, step by step

  1. Waves

    What we hear or see as a waveform is, at its simplest, a sine wave. Sound travels through a medium as compression and rarefaction: it is not the air molecules that travel, only regions of higher and lower pressure.

  2. Into the ear

    The pressure funnels down the ear canal and moves the eardrum, which vibrates in sympathy. The bones of the middle ear amplify the motion and pass it to the fluid-filled cochlea, where hair cells turn it into electrical impulses. From waves to bones to fluid to electricity: only when those impulses form patterns in the brain do we hear sound, and a few milliseconds later it becomes feeling. Sound is touch from a distance.

  3. Acoustics and psychoacoustics

    Each physical property has a perceived partner: amplitude is heard as loudness, frequency as pitch, and the pattern of vibration as timbre. Loudness is logarithmic, roughly twice as loud for every 10 dB, and it depends on frequency, as the equal-loudness contours first measured by Fletcher and Munson show. Timbre leads to masking, where one sound hides another.

  4. Consonance and dissonance

    A minor second rubs; a fifth settles. Familiar, recognisable sounds are accepted easily, while unfamiliar or dissonant ones keep the brain working to make sense of them. We are pattern-seeking listeners.

  5. When does sound become music?

    Loop a short spoken phrase and, after a few repetitions, many listeners start to hear it sung. Diana Deutsch called this the speech-to-song illusion. We play it, along with a scrambled melody that only becomes recognisable once you know what to listen for.

  6. Where is that sound?

    The Haas, or precedence, effect: the first sound to arrive fixes where we hear a source, even if later reflections are louder. Above about 1.5 kHz the head casts an acoustic shadow, so level differences between the ears help us localise. Sounds directly in front and directly behind give the same cues, a region called the cone of confusion, which we resolve by moving our heads. In a room, even a single reflection tells the brain about the space.

  7. Illusions

    The Shepard tone, a ‘sonic barber’s pole’ that rises forever. Deutsch’s scale and octave illusions, in which right-handers tend to hear higher tones on the right whatever the headphones play. And phantom words: ambiguous repeated sounds in which people hear words such as ‘window’, ‘welcome’, ‘rainbow’ or ‘mango’, often related to what is on their minds. The brain is constantly looking for meaning, even where there is none.

What we listen to

Each session uses auditory illusions to show that hearing is something the brain does, not just the ear. The Shepard tone, described by Roger Shepard in 1964, seems to rise forever without getting higher. Diana Deutsch’s octave illusion plays two alternating tones to the two ears, and most listeners hear a single tone hopping from ear to ear and changing pitch, though that is not what is played. With the ‘missing fundamental’, we hear a low pitch that is not physically present at all, which is how a small phone speaker can still carry a bass line.

These effects are the bridge to the technical half of the workshop. If perception has rules, those rules can be written down for a computer. Using Max/MSP, a visual programming environment that began with Miller Puckette’s Patcher at IRCAM in the late 1980s, participants build filters, analysers and simple listening machines that react to pitch, loudness and timing.

References

  1. Shepard, R. N. (1964). Circularity in judgments of relative pitch. Journal of the Acoustical Society of America, 36(12), 2346–2353.
  2. Deutsch, D. (1974). An auditory illusion. Nature, 251, 307–309. Link
  3. Deutsch, D. (2019). Musical Illusions and Phantom Words: How Music and Speech Unlock Mysteries of the Brain. Oxford University Press.
  4. Zwicker, E. & Fastl, H. (2007). Psychoacoustics: Facts and Models (3rd ed.). Springer.
  5. Puckette, M. (1988). The Patcher. Proceedings of the International Computer Music Conference, Cologne.
  6. Fletcher, H. & Munson, W. A. (1933). Loudness, its definition, measurement and calculation. Journal of the Acoustical Society of America, 5(2), 82–108.
  7. Haas, H. (1951). Über den Einfluss eines Einfachechos auf die Hörsamkeit von Sprache. Acustica, 1, 49–58.
  8. Deutsch, D. (1975). Two-channel listening to musical scales. Journal of the Acoustical Society of America, 57(5), 1156–1160.
  9. Deutsch, D., Henthorn, T. & Lapidis, R. (2011). Illusory transformation from speech to song. Journal of the Acoustical Society of America, 129(4), 2245–2252.
  10. Deutsch, D. Phantom words and other curiosities (audio demonstrations). Link

Have a space, a sound or an idea?

We design custom immersive experiences and tools for artists and businesses, and we’re always looking for spaces to listen to.

contact@sound.codes