1 of 10

Visualizing Digital Audio 

Beyond the waveform

​

Yotam Mann

2 of 10

Current Representations of Digital Audio

Show: 

  • Amplitude at a given time
  • Silence and breaks 

​

Lack:

  • Frequency and spectral information
  • Note attacks/decay
  • Beauty

​

​

3 of 10

Problem

  • The current representation of audio in Digital Audio Workstations (DAWs) does not provide sufficient information to music editors, sample-based music performers, and digital music composers. 
    • Waveforms all look the same, so it is hard to discern two different sounding samples.
    • It is difficult to see many sonic events if playing along to an audio file or arranging them.
    • Users have little sense of what the file sounds like by looking at it. 

4 of 10

Alternative Representations

  • Scores give pitch, duration and amplitude, but are, at the moment, impossible to generate by a computer for most audio. 
  • Spectrographs can be hard to parse, are often bland looking, and not incorporated into most DAWs. 

5 of 10

Alternative Representations Cont.

  • EQs don't let you see temporal events, they only show what is happening at a single moment. 
  • Music visualizers are usually too abstract and non-deterministic. 

6 of 10

Ligeti's Artikulation (1958)

Score by Rainer Wehinger

7 of 10

My Solution

  • Extract information from the FFT such as spectral centroid, spectral spread, approximate attacks, noisiness, amplitude, etc. 
  • Create a colorful and stylized visualization inspired by Wehinger's to display this information. 
  • Integrate the visualization into the DAW, Ableton Live. 

8 of 10

Expected Challenges

  • Creating a visual language to describe sound. 
    • I plan to make a legend similar to, but more general than Wehinger's with features mapped to appropriate shapes and colors.
  •  Aesthetics
    • Making the resulting image look like a hand-drawn score will likely require layout optimization. 
  • Integration into DAW
    • Ableton's new Max/MSP for Live should simplify this.

9 of 10

Game Plan   

  • Compile a list of necessary descriptors to characterize most sounds. 
  • Visually encode these descriptors in some meaningful way.
  • Extract these descriptors from audio.
  • Arrange the visual encodings so they match the audio well, and are visually pleasing. 
  • Integrate the visualization into Ableton Live.

10 of 10

Related Works

  • An Exploration of Real-Time Timbre Visualization by Kai Siedenburg (2009)
  • Zsa.descriptor a library for real-time descriptor analysis by Makhail Malt and Emmanuel Jourdan (2008)