1 of 71

2 of 71

👏 Thank you plug-in format developers 👏

2

3 of 71

Talk Structure

3

Changes to the plug-in formats

​

  1. Principles of good API design - No functional implications
    1. enforce correct usage by design (less bugs)
    2. easier for newcomers to learn�
  2. Limitations of the current API Design
    • Issues that make certain features impossible or very hard to implement

​

  • Biggest user-facing pain points

​

Developer facing

User facing

4 of 71

4

5 of 71

  1. Principles of good API design

5

a. Prefer state-less APIs

b. Identify Invariants

c. Single source of truth

d. Separation of concerns

e. Data-race free by design

​

b. Identify Invariants

c. Single source of truth

d. Separation of concerns

e. Data-race free by design

a. Prefer state-less APIs

6 of 71

Prefer state-less APIs

​

​

Let’s create an error window:

Background Colour (hexadecimal color notation)

6

State-less

AlertBox(640, 480, ColourSpace::cmyk, 0x30808080, "Hello World!", "OK");

​

​

​

​

7 of 71

Prefer state-less APIs

​

​

Let’s open a window:

Optional?

7

State-full

SetWidth(640);

SetHeight(480);

SetColourSpace(ColourSpace::cmyk);

SetBackgroundColour(0x30808080);

AlertMessage("Hello World!");

AlertButton("OK");

​

​

AlertBox();

8 of 71

Prefer state-less APIs

​

​

Let’s open a window:

Overrides or adds additional Button?

8

State-full

SetWidth(640);

SetHeight(480);

SetColourSpace(ColourSpace::cmyk);

SetBackgroundColour(0x30808080);

AlertMessage("Hello World!");

AlertButton("OK");

AlertButton("Cancel");

​

AlertBox();

​

9 of 71

Prefer state-less APIs

​

​

Let’s open a window:

Reorder: Is the color still interpreted in CMYK

9

State-full

SetWidth(640);

SetHeight(480);

SetBackgroundColour(0x30808080);

SetColourSpace(ColourSpace::cmyk);

AlertMessage("Hello World!");

AlertButton("OK");

AlertButton("Cancel");

​

AlertBox();

​

10 of 71

Prefer state-less APIs

​

​

Let’s open a window:

Does the state persist?

🡆 State-full APIs are bad

10

State-full

SetWidth(640);

SetHeight(480);

AlertMessage("Hello World!");

AlertButton("OK");

​

AlertBox();

​

AlertMessage("Bye World!");

AlertBox();

​

11 of 71

Prefer state-less APIs

​

​

AudioUnits:

11

​

​

​

​

​

​

​

​

​

​

​

[au allocateRenderResourcesAndReturnError:nullptr];

...

[au deallocateRenderResources];

​

​

​

​

​

auto* format = [[AVAudioFormat alloc] initStandardFormatWithSampleRate:48000.

channels:2] autorelease];

[au.inputBusses [0] setFormat:format error:nullptr];

[au.outputBusses[0] setFormat:format error:nullptr];

​

​

​

​

����

​

au.inputBusses [0].enabled = YES;

au.outputBusses[0].enabled = YES;

​

​

State-full

Order?

au.maximumFramesToRender = 512;

au.renderingOffline = false;

12 of 71

Prefer state-less APIs

​

​

12

State-less

void plugInActivate(int inChannels, int outChannels, double sampleRate,

int maxBufferSize, bool nonRealtime)

13 of 71

Prefer state-less APIs

​

​

13

State-less

void plugInActivate(ActivationParameters const& params);

struct ActivationParameters

{

int inputChannels, outputChannels;

double sampleRate;

int maxBufferSize;

bool nonRealtime;

};

14 of 71

Prefer state-less APIs

​

​

14

State-less

void plugInActivate(ActivationParameters const& params);

struct ActivationParameters

{

PlugInStatus setBusArrangements(bool isInput, int busIdx, ChannelLayout const& layout);

double sampleRate;

int maxBufferSize;

bool nonRealtime;

};

​

15 of 71

Prefer state-less APIs

​

​

15

State-less

void plugInActivate(ActivationParameters const& params);

​

​

​

​

​

​

​

​

struct ActivationParametersV2 : ActivationParameters

{...};

struct ActivationParameters

{

PlugInStatus setBusArrangements(bool isInput, int busIdx, ChannelLayout const& layout);

PlugInStatus setSampleRate(double sampleRate);

PlugInStatus setMaxBufferSize(int maxBufferSize);

PlugInStatus setNonRealtime(bool nonRealtime)

};

​

​

Isnt’t this state-full again?

No! It’s absolutely clear when these parameters become effective

16 of 71

  • Principles of good API design

16

a. Prefer state-less APIs

​

c. Single source of truth

d. Separation of concerns

e. Data-race free by design

b. Identify Invariants

17 of 71

Identifying Invariants

​

​

AudioUnits:

17

au.maximumFramesToRender = 512;

au.renderingOffline = false;

​

[au allocateRenderResourcesAndReturnError:nullptr];

​

​

au.renderBlock(nullptr, nullptr, 0, 0, audioBuffer, pullBlock);

[au deallocateRenderResources];

​

​

Many plug-in API methods will result in errors when called in an incorrect state

18 of 71

Identifying Invariants

AudioUnits:

Method:

18

au.currentPreset = coolPreset;

au.renderingOffline = false;

au.maximumFramesToRender = 512;

au.renderQuality = 64;

au.reset;

invoking au.renderBlock

[au.parameterTree parameterWithAddress:kGainParamID].value = 0.8f;

Allowed in:

✔

✗

✗

?

✔

✔

✔

Active State:

✔

✔

✔

✔

✗

✗

✔

In-Active State:

19 of 71

Identifying Invariants

​

​

​

​

19

How can the API help us call the correct methods in the correct state

20 of 71

Identifying Invariants

​

​

​

​

Clap:

Method:

Documentation

20

// Call start processing before processing.

// Returns true on success.

// [audio-thread & active & !processing]

bool(CLAP_ABI *start_processing)(const struct clap_plugin *plugin);

​

  • Unfortunately, the Plug-in format APIs don’t go beyond documentation
  • Wouldn’t it be great if calling the correct method in the correct state is enforced by the API?

​

21 of 71

Identifying Invariants

​

​

​

​

API enforces correct behaviour

21

State-less

void plugInActivate(ActivationParameters const& params);

struct ActivationParameters

{

PlugInStatus setBusArrangements(bool isInput, int busIdx, ChannelLayout const& layout);

PlugInStatus setSampleRate(double sampleRate);

PlugInStatus setMaxBufferSize(int maxBufferSize);

PlugInStatus setNonRealtime(bool nonRealtime)

};

​

struct ActivationParametersV2 : ActivationParameters

{...};

State-less API can enforce invariants

22 of 71

Identifying Invariants

​

​

​

​

Use RAII Deactivates PlugIn

22

void plugInActivate(ActivationParameters const& params);

​

std::unique_ptr<ProcessingPlugIn> plugInActivate(ActivationParameters const& params);

struct ProcessingPlugIn

{

~ProcessingPlugIn();

virtual void reset();

Virtual void processAudio(AudioSampleBuffer const&);

};

void plugInActivate(ActivationParameters const& params);

23 of 71

  • Principles of good API design

23

a. Prefer state-less APIs

b. Identify Invariants

​

d. Separation of concerns

e. Data-race free by design

c. Single source of truth

Parameters

24 of 71

Single source of truth

​

​

  • Who owns the plug-in parameters?

24

Plug-In

Host

Business Logic

getPlugInMetadata

saveState/loadState

Processor

processAudio

reset

UI

25 of 71

Single source of truth

​

​

​

AudioUnits/VST3/AAX/CLAP:

25

void process(AudioBuffer const& audio, EventList const& inParamChanges, EventList& outParamChanges)

Processor

Parameter Change Notifications to Processor

26 of 71

Single source of truth

​

​

​

​

AudioUnits:

26

Plug-In

Business Logic AUAudioUnit

-(void)setValue:(float)value

-(float)value

UI

AudioUnitViewController

Processor

AURenderBlock

Host

Registers listener

Registers listener

  • Looks the business logic owns the parameters…. right?

27 of 71

Single source of truth

​

​

​

​

AudioUnits:

27

Plug-In

Business Logic

AUAudioUnit

-(void)setValue:(float)value

-(float)value

UI

AudioUnitViewController

Processor

AURenderBlock

Host

Sends automation via EventList

Periodically sends parameter updates on Timer

  • “Business Logic” does not always have most up-to-date parameter values
  • Who owns the parameters?

Updates UI

28 of 71

Who owns the plug-in parameters?

​

​

AudioUnits:

28

Plug-In

Business Logic

AUAudioUnit

-(void)setValue:(float)value

-(float)value

UI

AudioUnitViewController

Processor

AURenderBlock

Host

Parameter Value callback via listener

Calls setValue

?

?

  • Who updates the processor? It depends on the host
  • Can cause inconsistencies in updating the parameter

29 of 71

Single source of truth

​

​

​

​

So who should be the single source of truth?

29

Plug-In

Business Logic

UI

Processor

Host

✗

✗

30 of 71

Single source of truth

​

​

​

​

CLAP/VST3

30

Plug-In

Business Logic

UI

Processor

Host

✗

✗

Single source of truth

31 of 71

Single source of truth

​

​

​

​

CLAP

31

Message Thread

Plug-In

Business Logic

UI

Processor

Host

Single source of truth

UI informs Processor

Host calls

process callback as usual

Processor returns changed params in out event list

Only has parameter metadata and getParamValue

​

No setValue methods

Audio Thread

Processor informs UI

32 of 71

Single source of truth

​

​

​

​

VST3

32

Audio Thread

Plug-In

Business Logic

UI

Processor

Host

Single source of truth

Host calls

process callback as usual

Message Thread

UI informs Host on message thread

  • Unfortunately, the VST3 documentation it’s not super clear that the processor “owns” parameters

getParamNormalized

setParamNormalized

33 of 71

  • Principles of good API design

33

a. Prefer state-less APIs

b. Identify Invariants

c. Single source of truth

​

e. Data-race free by design

d. Separation of concerns

34 of 71

Separation of Concerns

​

​

​

34

JUCE: Everything in one class (AudioProcessor)

virtual const String getName() const = 0

✔

✔

✔

Interface

Called hy host?

Called by plug-in?

Implemented by plug-in?

■ Enforced by API

virtual void prepareToPlay(…) = 0

✔

✔

💣

Interface

bool setBusesLayout(const BusesLayout&)

✔

✗

💣

Host Convenience Method

void addParameter (AudioProcessorParameter*)

💣

✗

✔

Plug-In Convenience Method

virtual void setNonRealtime (bool isNonRealtime) noexcept

✔

✔

✔/💣

Interface / Plug-In Convenience Overload

35 of 71

Separation of Concerns

​

​

35

AudioUnits

@interface AUAudioUnit : NSObject

​

@property (readonly) NSString *manufacturerName;

​

​

​

​

- (BOOL)allocateRenderResourcesAndReturnError:(NSError **)outError;

​

​

@property (readonly) BOOL renderResourcesAllocated;

​

​

​

@property (readonly) AUAudioUnitBusArray *inputBusses;

…

@end

Not a @protocol

  • No “pure virtual” methods
  • Each method must have a default implementation
  • Do these methods need to be overridden?
  • If yes, do we need to call the super method?
  • If no, who is the method for: for plug-in implementers or for hosts?

​

36 of 71

Separation of Concerns

​

​

36

AudioUnits

@interface AUAudioUnit : NSObject

/*! @property manufacturerName */

@property (readonly) NSString *manufacturerName;

​

/*! @method allocateRenderResourcesAndReturnError:

@discussion Hosts must call this before beginning to render.

Subclassers should call the superclass implementation. */

- (BOOL)allocateRenderResourcesAndReturnError:(NSError **)outError;

​

/*! @property renderResourcesAllocated*/

@property (readonly) BOOL renderResourcesAllocated;

​

/*! @property inputBusses

@discussion Subclassers must override this property's getter. */

@property (readonly) AUAudioUnitBusArray *inputBusses;

…

@end

Does this need to be overridden?

37 of 71

Separation of Concerns

​

​

37

AudioUnits

// Aspects of AUAudioUnit of interest only to subclassers.

@interface AUAudioUnit (AUAudioUnitImplementation)

/*! @method shouldChangeToFormat:forBus:

This is called when setting the format on an AUAudioUnitBus.

The AU can override this method to check before allowing a new format to be set on the bus.

If this method returns NO, then the new format will not be set on the bus.

The default implementation returns NO if the unit has renderResourcesAllocated, otherwise it results YES. */

- (BOOL)shouldChangeToFormat:(AVAudioFormat *)format forBus:(AUAudioUnitBus *)bus;

​

/*! @method setRenderResourcesAllocated:

In the base class implementation of allocateRenderResourcesAndReturnError:, the property renderResourcesAllocated is set to YES.

If allocateRenderResourcesAndReturnError: should fail in a subclass, subclassers must use this method to set renderResourcesAllocated to NO. */

- (void)setRenderResourcesAllocated:(BOOL)flag;

@end

​

Obj-c categories limits interface to plug-in implementors

Expected to be overridden

Expected to be invoked

38 of 71

Separation of Concerns

​

​

38

VST3

  • Interface layer is in it’s own directory: pluginterfaces
  • Various classes that plug-in implementers can sub-class from
  • Hosts (mostly) use the interface layer directly

CLAP

  • The main clap repo is only the interface layer
  • Clap-helpers repo contains classes from which plug-in implementers can sub-class from
  • Clap-helpers repo contains “proxy” classes which hosts can use to host plug-ins

39 of 71

  • Principles of good API design

39

a. Prefer state-less APIs

b. Identify Invariants

c. Single source of truth

d. Separation of concerns

e. Data-race free by design

40 of 71

Data-race free by design

40

How can API design help us make less data-races?

  • Good documentation

Message Thread

Audio Thread

Parameter Metadata

UI

Channel Layout negotiation

Audio and Parameter processing

Variable

Variable

Variable

Variable

State save/restore

💣

41 of 71

Data-race free by design

41

📖📖 Documentation 📖📖

CLAP

...

// [main-thread & active]

void(CLAP_ABI *deactivate)(const struct clap_plugin *plugin);

​

// Call start processing before processing.

// [audio-thread & active & !processing]

bool(CLAP_ABI *start_processing)(const struct clap_plugin *plugin);

​

// Call stop processing before sending the plugin to sleep.

// [audio-thread & active & processing]

void(CLAP_ABI *stop_processing)(const struct clap_plugin *plugin);

​

// Query an extension.

// The returned pointer is owned by the plugin.

// It is forbidden to call it before plugin->init().

// You can call it within plugin->init() call, and after.

// [thread-safe]

const void *(CLAP_ABI *get_extension)(const struct clap_plugin *plugin, const char *id);

...

👍 👍 👍

  • Better: API is structured along thread-boundaries and can help us avoid data-races

42 of 71

Data-race free by design

42

AUAudioUnit

Message Thread

Audio Thread

Parameter Metadata

State save/restore

UI

Channel Layout negotiation

Audio and Parameter processing

Interesting: AudioUnit

43 of 71

Data-race free by design

43

Message Thread

Audio Thread

Parameter Metadata

State save/restore

Obj-C/C++

UI

Channel Layout negotiation

Audio and Parameter processing

Interesting: AudioUnit

44 of 71

Data-race free by design

44

Audio Thread

Message Thread

AudioProcessor

PlugView

VstEditController

Parameter Metadata

State save/restore

UI

Channel Layout negotiation

Audio and Parameter processing

Better: VST3

  • Unfortunately, not many plug-ins make use of this separation

Messages

45 of 71

Data-race free by design

45

Message Thread

CPU

Accelerator Card

Audio Thread

AAX_CProcessProc

AAX_CEffectGUI

AAX_CEffectParameters

Parameter Metadata

State save/restore

UI

Channel Layout negotiation

Audio and Parameter processing

Best: AAX

Shared Struct

field1

field2

field3

buffer1

Access only via AAX SDK (data-race free)

46 of 71

2. Limitations of the current API Design

46

a. Negotiating a process buffer size

b. Plug-In sand-boxing (out-of-process)

c. Asynchronous plug-in processing

a. Negotiating a process buffer size

I don’t know the solutions to these issues

​

b. Plug-In sand-boxing (out-of-process)

c. Asynchronous plug-in processing

47 of 71

Negotiating a process bufferSize

AudioUnit

VST3

JUCE

CLAP

47

ProcessSetup::maxSamplesPerBlock = 512;

au.maximumFramesToRender = 512;

void prepareToPlay(double sampleRate, int maxExpectedSamples)

​

bool (*activate)(const clap_plugin*, double rate, uint32_t minFrames, uint32_t maxFrames);

48 of 71

Negotiating a process bufferSize

48

Why maxBufferSize?

  • Every plug-in for themselves should be able to determine which maximum buffer size makes sense for them
    • i.e. the buffer-size at which an additional increase in buffer-size leads to no significant improvement in performance in your DSP algorithm
  • They should allocate their buffers with this size in mind

auto const n = buffer.getNumSamples();

for (int max, pos = 0; pos < n; pos += max) {

max = jmin(n - pos, kMaxBufferSize);

processSlice(AudioSampleBuffer(buffer.getArrayOfWritePointers(),

buffer.getNumChannels(), pos, max));

}

…but happens if the host invokes the process method with a larger buffer?

Chop it up!

maxBufferSize

✗ ✗ ✗

49 of 71

Negotiating a process bufferSize

49

So what’s the problem?

  • Certain DSP algorithms require constant buffer-sizes: FFTs, partitioned convolution reverbs…
  • But plug-in formats only guarantee maximum buffer size
  • For some plug-ins today, the only way to workaround the plug-in format API limitation, is to add latency

50 of 71

Negotiating a process bufferSize

50

Possible Solution: DAW provides list of guaranteedBufferSizes

  • Most DAWs only call the process block with two different buffer-sizes
    • A smaller buffer size when the track is armed
    • A larger buffer size when it’s not

51 of 71

2. Limitations of the current API Design

51

a. Negotiating a process buffer size

​

c. Asynchronous plug-in processing

b. Plug-In sand-boxing (out-of-process)

52 of 71

Plug-In sand-boxing

52

Host’s process

In-process

Plug-In

UI

Audio and Parameter processing

Host

UI

Audio and Parameter processing

Message Thread

Audio Thread

53 of 71

Plug-In sand-boxing

53

Plug-In’s process

Host’s process

Out-of-process

Plug-In

UI

Audio and Parameter processing

Host

UI

Audio and Parameter processing

Message Thread

Audio Thread

Message Thread

Audio Thread

54 of 71

Plug-In sand-boxing

54

Time

Plug-In

① timerCallback fires and plug-in locks a mutex

② user clicks on “save” button in host. Mouse event is queued as message thread busy.

③ plug-in calls getTrackName

④ host retrieves track name

⑤ plug-in unlocks mutex and finished timerCallback

⑥ host’s mouse event handler is called

⑦ host calls plug-in’s getState

⑧ plug-in takes the same mutex and retrieves state

⑨ plug-in unlocks mutex and return

⑩ host finishes saving

Host

Message Thread

In-process

55 of 71

Plug-In sand-boxing

55

Message Thread

Message Thread

Time

Plug-In

Host

① timerCallback fires and plug-in locks a mutex

② user clicks on “save” button in host. Host’s event handler is called

④ plug-in calls getTrackName

③ host’s mouse event handler is called

⑤ host calls plug-in’s getState

⑥ plug-in tries to lock the same mutex and dead-locks

Host’s message thread is processing mouse event so plug-in needs to wait

Host can’t call into plug-in because message thread is processing timerCallback. To not deadlock host calls getState on aux thread

💣

Out-of-process

Aux Thread

56 of 71

2. Limitations of the current API Design

56

a. Negotiating a process buffer size

b. Plug-In sand-boxing (out-of-process)

c. Asynchronous plug-in processing

57 of 71

Asynchronous plug-in processing

57

void processAudio(AudioBuffer& buffer)

  • The humble processAudio callback

void processAudioAsync(AudioBuffer const& inBuffer, Timestamp deadline,

std::function<void (AudioBuffer<float> const& outBuffer)> && callback)

​

  • Asynchronous processAudio call?

Why?

Blocks until all audio is processed

58 of 71

Asynchronous plug-in processing

58

512smpls = ~ 10.6ms

CPU

80%

Bling PlugIn Processor

User buys the “Bling Accelerator Card”

512smpls = ~ 10.6ms

Computationally demanding plug-in

​

Blocks the audio for 8ms

​

Host calls blocking

processAudio

processAudio

callback finishes and passes control back to host

59 of 71

Asynchronous plug-in processing

59

Bling Accelerator Card

Computationally demanding plug-in

​

Blocks the audio for 8ms

​

CPU

80%

  • No way to solve this problem without adding a buffer of latency

Bling PlugIn Processor

Host calls blocking

processAudio

512ms = ~ 10.6ms

processAudio

callback finishes and passes control back to host

Passes audio to accelerator

Accelerator passes processed audio back to plugin

😴

😴

  • Problem also occurs if audio sent to other apps for processing

512ms = ~ 10.6ms

60 of 71

3. Biggest user-facing pain points

60

  1. Plug-in scanning

b) Missing plug-ins

c) Copy protection

61 of 71

Biggest user-facing pain points

61

Why is plug-in scanning necessary?

  1. Collect metadata about plug-ins
    1. Name, Version, Manufacturer
    2. Channel configurations
    3. UI previews
    4. …�
  2. Plug-in verification
    • You don’t want plug-ins crashing the DAW
    • Even when loading plug-ins out-of-process, plug-ins can still misbehave and cause issues: dead-locks

62 of 71

Biggest user-facing pain points

62

Why is plug-in scanning necessary?

  • Solution for collecting plug-in metadata

VST3:

  • moduleinfo.json containing metadata is part of the plug-in’s vst3 bundle
  • Can be automatically generated while the plug-in bundle is being built by the developer
  • “Developer” scans the plug-in once, instead of the user
  • Optional
  • Does not contain all the metadata that various hosts need
  • Still requires scanning to verify the plug-in’s will not crash the host

63 of 71

Biggest user-facing pain points

63

Online plug-in scanning tool for plug-in developers

Developer creates new plug-in

Developer uploads plug-in to plug-in verification service

Verification service gathers metadata verifies plug-in on various OSes/DAWs

Developer receives detailed report on failure

✗

On success, verification service add metadata to plug-in package and cryptographically signs it

Developer distributes signed plug-in to users

  • Signature now guarantees that the plug-in has been successfully verified

64 of 71

3. Biggest user-facing pain points

64

  • Plug-in scanning

b) Missing plug-ins

c) Copy protection

65 of 71

Biggest user-facing pain points

65

Online track rendering

Verification/render service has copy of the plug-in and can render track and generate a movie of plug-in’s GUI responding to automation

User opens project file with missing plug-ins

DAW uploads project to render/verification service

DAW uses pre-rendered audio. User can see automation affecting UI but parameters/automation disabled.

Plug-in Missing

66 of 71

3. Biggest user-facing pain points

66

  • Plug-in scanning

b) Missing plug-ins

c) Copy protection

67 of 71

Biggest user-facing pain points

67

Copy Protection

  • There needs to be a single standard for how DAWs/Plug-Ins integrate various copy protection solutions
  • Independent of the actual copy protection solution being used
  • At the very minimum: Plug-Ins should communicate to the DAW if their licence check failed
    • Instead of each individual unlicensed plug-in blocking the scanning progress with random dialog boxes

68 of 71

Biggest user-facing pain points

68

  • Plug-in scanning

b) Missing plug-ins

c) Copy protection

All three solutions require a trusted entity!

Who should it be?

69 of 71

Biggest user-facing pain points

69

  • Single entity trusted by both DAW and plug-in vendors
  • Offers plug-in online verification/rendering service (open-source!)
  • Offers assistance on standardization: for example, copy protection
  • Should be plug-in format independent
  • Elected governing structure (½ DAW vendors, ½ plug-in vendors)
  • Collects small fee for verification-process
  • Non-profit

Audio Plug-in Industry Association

70 of 71

Biggest user-facing pain points

70

Closing Remarks

  • I love the audio plug-in community
    • Many diverse independent developers
    • Not many industries like that these days

​

  • If we don’t solve this problem soon, some big company will
  • It will be proprietary and expensive
  • Convenience is king: if someone offers a solution, users will prefer that solution
  • and it might mean the end to the plug-in industry as we love it

Who should run the verification/render service?

71 of 71

71

Thank you!�

Questions?