1 of 101

2 of 101

About me

Simon HENRY

2D animation supervisor at CARIBARA Angouleme

worked on two Blender projects :

creator of sound2mesh :

3 of 101

Admo - Aftershock

Amusement - SSKGCCULT

Calm Down - Chris Jones

finding ways to visualize sound

The Idea

(youtube examples)

4 of 101

The Idea

MIDI visualization

“Fast Fourier Transform” tests

(inspired by 3blue1brown)

finding ways to visualize sound

(personal work)

5 of 101

Summary

Spectrogram ?
Prototype

spectrogram
waveform
shader
having fun

sound2mesh

FFmpeg to the rescue
precision control
artistic control
User Interface

Tips

6 of 101

Spectrogram ?

.mp3 to .png

7 of 101

“A spectrogram is a visual representation

of the [...] frequencies of a signal”

- Wikipedia

Spectrogram ?

8 of 101

Spectrogram ?

waveform

spectrogram

9 of 101

Spectrogram ?

Time

waveform

Amplitude

Time

spectrogram

Frequency

Amplitude

10 of 101

11 of 101

Prototype

moving vertices

12 of 101

Prototype (spectrogram)

base grid

displacement

texture

13 of 101

Prototype (spectrogram)

UV scroll

14 of 101

Waveform

immediate response

15 of 101

Prototype (waveform)

spectrogram

spectrogram + waveform

16 of 101

17 of 101

Shader

color narrative

18 of 101

Prototype (shaders)

base texture

19 of 101

Prototype (shaders)

base texture

height gradient

20 of 101

Prototype (shaders)

base texture

height gradient

custom

21 of 101

Prototype (shaders)

22 of 101

Prototype (shaders)

slopes

valleys

ridges

23 of 101

Prototype (shaders)

24 of 101

Having Fun

♫♪

25 of 101

Having Fun

♪♫ Fruko y Orquesta - Mambo No. 5 (1993)

26 of 101

Having Fun

♪♫ Digging My Potato - The Seatbelts (1998)

27 of 101

Having Fun

♪♫ Cats On Mars - Yōko Kanno (1998)

28 of 101

Having Fun

♪♫ Daft Punk - Touch (2013)

29 of 101

Digging My Potato - The Seatbelts (1998)

close-up view

Having Fun

30 of 101

Having Fun (shape)

arc

31 of 101

Having Fun (shape)

tube (exterior)

32 of 101

Having Fun (shape)

tube (interior)

33 of 101

Having Fun (shape)

camera view

34 of 101

Retrowave

♪♫ Plastic Love - Mariya Takeuchi

35 of 101

Disco

♪♫ Holding Out for a Hero - Jennifer Saunders

36 of 101

Mountains

♪♫ Riding (Day) - The Legend of Zelda: Breath of the Wild

37 of 101

Neon

♪♫ Touch - Daft Punk

38 of 101

Nebula

♪♫ Alfonsina y èl Mar - Rita Payès

39 of 101

sound2mesh

part 1 : FFmpeg to the rescue

40 of 101

Generate spectrogram (Audioalter)

https://audioalter.com/spectrogram

41 of 101

Generate spectrogram (FFmpeg)

https://ffmpeg.org/

42 of 101

Generate spectrogram (FFmpeg)

https://xkcd.com/2347/

43 of 101

Generate spectrogram (FFmpeg)

[…]

showcqt()

showcwt()

showfreqs()

showspectrum()

showvolume()

showwaves

showwavespic

[…]

showspectrumpic()

44 of 101

FFmpeg - showspectrumpic

showspectrumpic(color, limit, frange)

spectrogram color

volume limit

frequency range

45 of 101

FFmpeg - showspectrumpic

showspectrumpic(color, limit, frange)

spectrogram color

46 of 101

FFmpeg - showspectrumpic

channel

cividis

cool

fiery

fire

fruit

green

intensity

magma

moreland

nebulae

plasma

rainbow

terrain

viridis

FFmpeg color gradients

47 of 101

FFmpeg - showspectrumpic

image

luminance

displacement

Blender uses the -luminance -

to calculate displacement

(no matter the hue)

48 of 101

FFmpeg - showspectrumpic

spectrogram

luminance

displacement

Blender uses the -luminance -

to calculate displacement

(no matter the hue)

49 of 101

FFmpeg - showspectrumpic

channel

cividis

cool

fiery

fire

fruit

green

intensity

magma

moreland

nebulae

plasma

rainbow

terrain

viridis

FFmpeg color gradients

50 of 101

FFmpeg - showspectrumpic

showspectrumpic(color, limit, frange)

volume limit

51 of 101

-48 dB

-24 dB

0 dB

audio

spectrogram

FFmpeg - showspectrumpic

faint sound

=

low brightness

loud sound

=

high brightness

52 of 101

FFmpeg - showspectrumpic

without compensation

showspectrumpic()

with

compensation

ffvolumedetect()

showspectrumpic()

use ffvolumedetect() to get the �max volume of the sound

feed the value into showspectrumpic()-

to always get max brightness

53 of 101

FFmpeg - showspectrumpic

-48 dB

-24 dB

0 dB

audio

spectrogram

(with compensation)

same brightness all around !

54 of 101

FFmpeg - showspectrumpic

showspectrumpic(color, limit, frange)

frequency range

55 of 101

FFmpeg - showspectrumpic

lin

log

(linear)

(logarithmic)

56 of 101

FFmpeg - showspectrumpic

lin

log

(linear)

(logarithmic)

57 of 101

FFmpeg - showspectrumpic

github.com/FFmpeg/FFmpeg/blob/master/libavfilter/avf_showspectrum.c

58 of 101

FFmpeg - showspectrumpic

github.com/FFmpeg/FFmpeg/blob/master/libavfilter/avf_showspectrum.c

59 of 101

FFmpeg - showspectrumpic

60 of 101

FFmpeg - showspectrumpic

Uses the formula to -distort UVs-

Can interpolate between lin and log -from a single image-

61 of 101

FFmpeg - showspectrumpic

linear

logarithmic

Allows for both -precise-

and -artistic- control !

62 of 101

Precision Control

0.000000000000001

63 of 101