AudioWorklet
:: What, Why and How
Hongchan Choi | Google Chrome
“What is AudioWorklet?”
“What is Worklet?”
"...API for running scripts in stages of the rendering pipeline
independent of the main javascript execution environment..."
“What is Worklet?”
"...API for running scripts in stages of the rendering pipeline
independent of the main javascript execution environment..."
Main Execution Context
Rendering Pipeline
worklet.addModule('script.js')
script.js
"Main Global Scope"
"Main Thread"
"Worklet Global Scope"
"Rendering Thread"
module
loading
Worklet
PaintWorklet
AudioWorklet
AnimationWorklet
Web Audio API
(WebAudio render thread)
CSS Paint API�(Main thread)
CSS Animation Worklet API
(Compositor thread)
“What is AudioWorklet?”
"...exposing the low-level audio processing capability�to web developers within Web Audio API's ecosystem."
"What ScriptProcessor?”
ScriptProcessorNode : PROBLEMS
Underspecified.
Double buffering.
Access to DOM.
Runs on main thread.
Data race: glitch, drop-out, stuttering
“Well, is it that bad?”
BiquadFilter
Node
Gain�Node
Destination
Node
BufferSource
Node
Main Thread�(Global Scope)
BiquadFilter
Node
Gain�Node
Destination
Node
BufferSource
Node
Main Thread
WebAudio Render Thread
BiquadFilter
Internal
Gain
Internal
Destination
Internal
BufferSource
Internal
ScriptProcessor
Node
Destination
Node
BufferSource
Node
Main Thread
WebAudio Render Thread
ScriptProcessor
Internal
Destination
Internal
BufferSource
Internal
processor
onaudioprocess
(script code)
async event handling
“Well, how is AWN better?”
AudioWorklet�Node
Destination
Node
BufferSource
Node
Main Thread
WebAudio Render Thread
AudioWorklet�Internal
Destination
Internal
BufferSource
Internal
AudioWorklet�Processor
(script code)
JS
Engine
(e.g. V8)
AudioWorklet : BENEFITS
Well-defined.
JS code as a first class citizen.
Native support on AudioParam.
Runs on WebAudio render thread.
Extensibility.
and so on...
“Okay, how do I use it?”
AudioWorklet
AudioWorkletGlobalScope
AudioWorkletNode
AudioWorkletProcessor
audioWorklet.addModule('bypass.js').then(() => {
let context = new AudioContext();
let oscillatorNode =
new OscillatorNode(context);
let bypassNode =
new AudioWorkletNode(context, 'bypass');
oscillatorNode
.connect(bypassNode)
.connect(context.destination);
});
class Bypass extends AudioWorkletProcessor {
constructor() {
super();
}
process(inputs, outputs) {
/** process audio **/
}
}
registerProcessor('bypass', Bypass);
"bypass.js"
Main Global Scope
AudioWorkletGlobalScope
process(inputs, outputs) {
let input = inputs[0];
let output = outputs[0];
for (let channel = 0; channel < input.length; ++channel) {
output[channel].set(input[channel]);
}
return true;
}
Bypass.process()
AudioWorkletGlobalScope
sequence<sequence<Float32Array>>
AudioWorklet�Processor
Input #0�4 Channels
Input #1�3 Channels
Output #0�2 Channels
Output #1�3 Channels
inputs[0][2]
outputs[1][0]
inputs
outputs
Input #2�1 Channel
AudioParam
❤
AudioWorklet
audioWorklet.addModule('bypass.js').then(() => {
let context = new AudioContext();
let bypassNode =
new AudioWorkletNode(context, 'bypass');
let gainParam =
bypassNode.parameters.get('gain');
gainParam.value = 1.0;
gainParam.linearRampToValue(0.0, 5.0);
});
class Bypass extends AudioWorkletProcessor {
static get parameterDescriptors() {
return [{
name: 'gain',
defaultValue: 0.0
}];
}
constructor() {...}
process(inputs, outputs, parameters) {...}
}
registerProcessor('bypass', Bypass);
"bypass.js"
Main Global Scope
AudioWorkletGlobalScope
AudioWorklet�Processor
"gain" parameter�values
inputs
outputs
parameters
AudioParam�Processing
(e.g. automation)
process(inputs, outputs, parameters) {
let input = inputs[0];
let output = outputs[0];
let gainValues = parameters['gain'];
for (let channel = 0; channel < input.length; ++channel) {
let inputChannel = input[channel];
let outputChannel = output[channel];
for (let i = 0; i < inputChannel.length; ++i)
outputChannel[i] = inputChannel[i] * gainValues[i];
}
return true;
}
Bypass.process()
AudioWorkletGlobalScope
MessagePort
❤
AudioWorklet
AudioWorklet�Node
AudioWorklet
Processor
.port
.port
<examples>
myCompressor.type = 'hardknee';
mySource.startNote(time);
<examples>
this.sendFFTResult();
this.onended();
audioWorklet.addModule('dummy.js').then(() => {
let context = new AudioContext();
let dummyNode =
new AudioWorkletNode(context, 'dummy');
dummyNode.port.onmessage = (event) => {
console.log(event.data);
dummyNode.port.postMessage('backatya!');
};
});
class Dummy extends AudioWorkletProcessor {
constructor() {
super();
this.port.postMessage('ready!');
this.port.onmessage = (event) => {
console.log(event.data);
};
}
process(inputs, outputs, parameters) {...}
}
registerProcessor('dummy', Dummy);
"dummy.js"
Main Global Scope
AudioWorkletGlobalScope
(real) CurrentTime
❤
AudioWorklet
audioWorklet.addModule('dummy.js').then(() => {
let context = new AudioContext();
let dummyNode =
new AudioWorkletNode(context, 'dummy');
});
class Dummy extends AudioWorkletProcessor {
constructor() {
super();
this.createdAt_ = currentTime;
}
process(inputs, outputs, parameters) {
if (currentTime - this.createdAt_ > 5.0) {
/* do something here */
}
}
}
registerProcessor('dummy', Dummy);
"dummy.js"
Main Global Scope
AudioWorkletGlobalScope
“Sure, can I use it now?”
Available
in Chrome 66
Origin Trial
in Chrome 64, 65
1. Sign Up for Origin Trial.
For Chrome 64 and 65
$CHROME_CANARY_PATH --enable-blink-features=AudioWorklet
2. Use Command Line Option.
(recommended)
For Chrome 64 and 65
3. Use "Flags" setting.
Enable "Experimental Web Platform Features" and
re-launch the browser.
(Not recommended)
For Chrome 64 and 65
AudioWorklet Demo
“Where do we go from here?”
Possibilities
WASM
Shared array buffer (disabled due to spectre/meltdown)
AudioWorklet module distribution
Dynamic code generation
and so on...
Questions?
@hochsays | github.com/hoch
Backup Material
For more WebAudio fun!
processor
processor
onaudioprocess
(script code)
1
2
1
2
onaudioprocess
(script code)
write
read
write
read
dispatch
event
dispatch
event
ScriptProcessorNode : Data Race
processor
processor
onaudioprocess�(script code)
1
2
write
read
dispatch
event
1
2
read
still writing?
ScriptProcessorNode : Data Race
!