WebAudio tutorial�Audio Effects and Musical Instruments in the Browser
TheWebConference 2018, Lyon France
W3C Track,
Michel Buffa,
Université Côte d’Azur, France,
I3S/CNRS/INRIA labs
buffa@i3s.unice.fr, @micbuffa
�
Stéphane Letz letz@grame.fr,
Yann Orlarey orlarey@grame.fr
GRAME/Lyon
Some WebAudio examples are detailed in this MOOC at W3Cx.org
Outline
History
2010/2011: Audio Data API (Mozilla)
2010: proposal for a WebAudio API and discussion about its design concepts
15/12/2011: first draft specification of the WebAudio API
2012: implementations in Firefox/Webkit browsers
End of 2012 :
2018: AudioWorklet node implemented in Google Chrome
WebAudio Concepts
Audio operations occur inside an Audio Context
let ctx = new AudioContext();
Modular design:
let osc1 = ctx.createOscillator();
osc1.frequency.value = 440;
let gain1 = ctx.createGain();
gain1.gain.value = 0.1;
osc1.connect(gain1).connect(ctx.destination);
WebAudio comes with a set of standard nodes
WebAudio provides a set of standard nodes:
Why high-level nodes?
Advantages:
Disadvantages:
Audio source nodes: AudioBufferSourceNode
Audio source nodes: OscillatorNode
Audio source nodes: AudioBufferSourceNode
Good utility: an asset loader (images, sounds)
Scheduling / playing sounds at a given time
sourceNode.start( time );�sourceNode.stop( time + delay );
Audio source nodes: MediaElementSourceNode
<audio id="player" src="http://.../guitarRiff1.mp3" controls ></audio>
ctx = new AudioContext();
var player = document.querySelector("#player");
var source = ctx.createMediaElementSource(player);
source.connect(ctx.destination);
Click on images to see the code
Audio source nodes: MediaStreamAudioSourceNode
navigator.mediaDevices.getUserMedia({audio: true})
.then((stream) => {
sourceNode = ctx.createMediaStreamSource(stream);
// build the audio graph
sourceNode.connect(...);
});
Audio destinations
var ctx = new AudioContext();
var osc = ctx.createOscillator();
var dest = ctx.createMediaStreamDestination();
var mediaRecorder = new MediaRecorder(dest.stream);
osc.connect(dest);
mediaRecorder.start(); // starts recording...
Let’s look at a first, simple example (GainNode)
An HTML audio player
With a GainNode connected to it
The GainNode is connected to ctx.destination (speakers)
var g = ctx.createGain();
source.connect(g);
g.connect(ctx.destination);
g.gain.value = 3; // louder x 3!
After volume, add stereo / balance control
An HTML audio player
With a StereoPanner node connected to it
The StereoPanner is connected to ctx.destination (speakers)
var p = ctx.createStereoPanner();
source.connect(p);
p.connect(ctx.destination);
p.pan.value = 0.8; // right > left
Control frequencies: filters
WebAudio comes with two types of filters:
var f = ctx.createBiquadFilter();
source.connect(f);
f.connect(ctx.destination);
f.type = "lowpass";
f.frequency.value = 200; // Hz
f.gain=-6; // dB
Control frequencies: filters, a multiband equalizer
Step by step tutorial available on the free MOOC by W3Cx: HTML5 Apps and Games, Week 1.
Let’s give it a look!
Add reverberation and other convolution effects
An HTML audio player with a ConvolverNode connected to it + 2 GainNode to make a “wet” and a “dry” route to ctx.destination (speakers). The slider adjusts the two gains, setting the amount of reverb we add to the signal.
Avoid clipping using a compressor node
An HTML audio player to a gain connected to a DynamicCompressorNode for avoiding clipping / saturation. Other example = multiple sound sample played at the same time needs to be limited.
Distortion: using WaveShaper nodes
An HTML audio player
With a WaveShaperNode connected to it�The WaveShaperNode is connected to ctx.destination (speakers)�Wave shapers use a “transfer function curve”.
var ws = ctx.createWaveShaper();
source.connect(ws);
ws.connect(ctx.destination);
// k = “squarifying” param
ws.curve = makeDistortionCurve(k);
Make “echo” effects: use a delay node!
Analyser node: visualize waveforms, frequencies, volume
Demos: effects and instruments, only using high level nodes
Conclusion of part 1: high-level WebAudio in JS...
Custom programmed nodes : why ?
Custom programmed nodes : how ?
Developed in pure JavaScript, or produced by another mean.
WebAssembly(wasm) is now the preferable way to deploy fast code in the Web:
ScriptProcessorNode/AudioWorkletNode
The AudioWorkletNode/AudioWorkletProcessor model (1)
The AudioWorkletNode/AudioWorkletProcessor model (2)
The AudioWorkletNode/AudioWorkletProcessor model (3)
The AudioWorkletNode/AudioWorkletProcessor model (4)
The AudioWorkletNode/AudioWorkletProcessor model (5)
MessagePort to communicate between Node/Processor
Demo of custom Noise node
Using Faust Domain Specific Language
Faust is a DSP for audio programming : effect, synthesis, instrument design…
Faust tutorial used for the demo: https://faust.grame.fr/tutorial/
Deploying Faust coded WebAudio nodes
Emscripten generated code
Now let’s discuss what is still missing...
First, let’s look at some computer music history….
At the industry…
At the “programming model”...
The Electronic Music landscape
The open source Computer Music landscape
Note: the only “professional-quality” DAW on Linux is commercial and not Open Source (Reaper)
WebAudio and Audio on the Web
Did you really listen to what we said earlier?
But… no plugin standard, no “hosts”, no programming model...
We find some very good JavaScript libraries (i.e. toneJS)
Some open source github repositories (i.e. https://webaudiodemos.appspot.com/)
Some online tools for music synthesis (genish.js etc.)
Some DSL for DSP programming (FAUST)
Some effects and instruments
With some researchers and developers we decided to start working on an open plugin standard for WebAudio
August 2017, WebAudio conference, Jari Kleimola from WAMs saw my presentation and contacted me...
We made a team with different researchers / developers, that share same concerns with different approaches
Audio on the Web vs Native audio
An open standard = API/specification ? Or more… ?
So… where how do we start working on a standard???
DEMOS
Simple example
DEMOS
Our world domination plan part 1
Native plugins (C/C++) written as VST, JUCE etc.
FAUST
WebAudio Plugins
(WAPs)
WebAudioModules
(WAMs)
�JavaScript
WebAssembly�+� AudioWorklet
Max DSP, Pure Data
Others...
Web browser
Our world domination plan part 2 (...)
WebAudio Plugins
(WAPs)
WebAudio host runs in a Web browser
Wrap embedded browser in a native plugin shell
Now the browser runs
as a native plugin
Load any WebAudio plugin
in native hosts, no need to rewrite GUI or DSP code