1 of 44

with Azure Cognitive Services �and Ably Realtime

WEARABLE �LIVE CAPTIONS

#1c4966

2 of 44

Hello � Build� Stuff!

Jo Franchetti

DevRel at Ably

@thisisjofrank

@ablyrealtime

#1c4966

3 of 44

MDD

Momma Franchetti

Mum Driven Design

#1c4966

This is my mum. Mumma Franchetti.

She's a smiley, sociable, quick witted. She loves a good natter with her mates.

She's also been steadily losing her hearing over the last few years. Too many rock concerts when she was young.

Losing her hearing has been an incredibly isolating experience for her, and I'm sure for many others. She relies heavily on lip reading and clear enunciation.

You know what makes talking clearly and lip reading impossible? Mask wearing.

If she was struggling to keep up with conversion before, Corona has made it almost impossible for her.

She can't socialise safely with friends, she can't understand people in shops.

People become rude and brush her off when she asks them to repeat themselves, or she gets self conscious and tries to style out not having heard what people said.

I have a couple of asks from you all in this talk, and the first one is this. If someone asks you to repeat yourself, don't respond with “never mind, or don't worry”. It's frustrating as hell being in on only half a conversation.

As we've traversed her hearing loss, one thing has really struck me, and that's that as developers we're really not doing a good job at making assistive technology.

My mum has a cochlear implant. It's pretty cool. A device inserted into her head. Cyborg mum!

It has a processor and a little battery that clips over her ear. This device can stream sounds from her phone directly into her head. No need for headphones! But the app that comes with it only works on iPhone. Why?! And you know what happened a few years ago when iOS released an update? Everyone with this particular brand of implant had constant screaming noises streamed directly into their heads for a day until a fix was rolled out. We can and have to do better than this.

I know assistive technologies aren't very sexy and so they don't get the attention that the glamorous software eng jobs get, but goodness, I hope we're better than this.

So my second ask of this talk is. Let's put some effort into accessibility and assistive tech and actually consider our fellow humans.

So, storytime is almost over, I promise. I was worried for my mum. She felt even more isolated than the rest of us did during lockdown. And assistive technologies that run on her phone meant that she had to be looking down at her phone throughout her conversations, which often lead to people ignoring her.

I'm a developer, I like to fix problems for people. I should be able to fix this problem for her. And inspiration struck me at the weirdest of places.

4 of 44

INSPIRATION STRUCK!

At Cyberdog in Camden of all places

#1c4966

5 of 44

INSPIRATION STRUCK!

At Cyberdog in Camden of all places

#1c4966

6 of 44

INSPIRATION STRUCK!

At Cyberdog in Camden of all places

#1c4966

7 of 44

LET’S BUILD IT!

But what will we need?

#1c4966

8 of 44

DISPLAY

MICROPHONE

A small, flexible array of addressable LEDs

SPEECH TO TEXT

IOT PROTOCOL

A device to pick up speech

Software to process the audio and convert it into text

Data transfer between the software and the display

#1c4966

9 of 44

THE DISPLAY

A 28 x 8 array of neopixels (very small addressable LEDs)
Powered by a rechargeable USB battery
Controlled by an ESP8266 microcontroller

#1c4966

10 of 44

THE ESP8266

Measures 5cm x 2cm
80MHz processor with 3.3V logic/power
USB Power
WiFi on board

#1c4966

11 of 44

THE MICROPHONE

Everyone has one

We need to process the data, which is a little intensive for the microprocessor

Why not use the one on our phones?

#1c4966

12 of 44

SPEECH �TO TEXT

WEB

APP

+

AZURE

COGNITIVE

SERVICES

Build a web app with HTML, CSS and JS which will run on a phone or computer

Requires an internet connection and a microphone

Use Web APIs to get data from the mic

Send to Azure Cognitive Services to get text back

https://azure.microsoft.com/en-us/services/cognitive-services/speech-to-text/

#1c4966

13 of 44

NODE JS APP

LED MATRIX DRIVER

ACS SPEECH SDK

REALTIME MESSAGING

THE WEB APP

https://github.com/ably-labs�/live-caption-demo

https://npmjs.com/package�/@snakemode/matrix-driver

https://npmjs.com/package/microsoft�-cognitiveservices-speech-sdk

https://npmjs.com/package/ably

#1c4966

14 of 44

#1c4966

15 of 44

USING ACS SPEECH SDK

public async streamSpeechFromBrowser() {

const speechConfig = await this.getConfig();

const audioConfig = SpeechSDK.AudioConfig.fromDefaultMicrophoneInput();

const recognizer = new SpeechSDK.SpeechRecognizer(speechConfig, audioConfig);

recognizer.recognized = async (_, speechResult) => {

const text = speechResult.privResult.privText ?? "";

this._callback(text);

};

recognizer.startContinuousRecognitionAsync(

function () { },

function (error) { console.log(error); }

);

}

#1c4966

16 of 44

USING ACS SPEECH SDK

public async streamSpeechFromBrowser() {

const speechConfig = await this.getConfig();

const audioConfig = SpeechSDK.AudioConfig.fromDefaultMicrophoneInput();

const recognizer = new SpeechSDK.SpeechRecognizer(speechConfig, audioConfig);

recognizer.recognized = async (_, speechResult) => {

const text = speechResult.privResult.privText ?? "";

this._callback(text);

};

recognizer.startContinuousRecognitionAsync(

function () { },

function (error) { console.log(error); }

);

}

#1c4966

17 of 44

USING ACS SPEECH SDK

const speech = new AzureCognitiveSpeech("/api/createAzureTokenRequest");

speech.onTextRecognised((text) => {

ledDriver.text.scroll(text, colorValue, speedValue);

});

#1c4966

18 of 44

SENDING DATA TO THE BOARD

MQTT is the protocol designed for IoT devices
It is lightweight and high latency
To use in the web we require an MQTT broker
For this project we’ll use Ably’s MQTT broker

+

#1c4966

19 of 44

THE ESP8266

Measures 5cm x 2cm
80MHz processor with 3.3V logic/power
USB Power
WiFi on board

#1c4966

The microprocessor is very small and therefore has limited processing power and memory, which means that the code that runs on it needs to be as efficient as possible.

So we really want to avoid doing complicated text parsing or other similarly memory intensive tasks on the microprocessor.

Parsing and splitting strings is especially costly, and involves larger buffers than would be ideal.

While at a glance, this may seem like premature performance optimisation, if all of the memory on the board is used parsing the messages as human readable strings, it decreases the amount of memory available to buffer incoming messages.

To solve this problem, I used a binary message format to talk to the hardware.

The browser app creates a specially coded message to send text strings to the device.

20 of 44

AN EXAMPLE COMMAND

const textMessage = {

value: "My line of text",

mode: 1,

scrollSpeedMs: 25,

color: { r: 255, g: 255, b: 255 }

};

#1c4966

21 of 44

AN EXAMPLE COMMAND

0	0x11	ASCII DC1 - Device Control 1
1	0x54	The ASCII character T - “text mode”
2	0x01	The value 1 - “scrolling text”
3	0x00-0xFF	The scroll interval, in ms. A number between 0-255 in hexadecimal
4	0x00-0xFF	The R component of the RGB text colour
5	0x00-0xFF	The G component of the RGB text colour
6	0x00-0xFF	The B component of the RGB text colour
7	0x02	ASCII STX - “start of text”
8	0x02-0x7E	The ASCII code for each character
9	0x03	ASCII ETX - “end of text”
10	0x04	ASCII EOT - “end of transmission”

#1c4966

22 of 44

AN EXAMPLE COMMAND

0	0x11	ASCII DC1 - Device Control 1
1	0x54	The ASCII character T - “text mode”
2	0x01	The value 1 - “scrolling text”
3	0x00-0xFF	The scroll interval, in ms. A number between 0-255 in hexadecimal
4	0x00-0xFF	The R component of the RGB text colour
5	0x00-0xFF	The G component of the RGB text colour
6	0x00-0xFF	The B component of the RGB text colour
7	0x02	ASCII STX - “start of text”
8	0x02-0x7E	The ASCII code for each character
9	0x03	ASCII ETX - “end of text”
10	0x04	ASCII EOT - “end of transmission”

#1c4966

23 of 44

THE ABLY MQTT BROKER

const ablyClient = new Ably.Realtime.Promise({ authUrl: "/api/createTokenRequest" });

const ledDriver = new RemoteMatrixLedDriver({

displayConfig: { width: 28, height: 8 },

deviceAdapter: new ArduinoDeviceAdapter([new AblyTransport(ablyClient)])

});

#1c4966

24 of 44

THE ABLY MQTT BROKER

const ablyClient = new Ably.Realtime.Promise({ authUrl: "/api/createTokenRequest" });

const ledDriver = new RemoteMatrixLedDriver({

displayConfig: { width: 28, height: 8 },

deviceAdapter: new ArduinoDeviceAdapter([new AblyTransport(ablyClient)])

});

#1c4966

25 of 44

THE ENTIRE SYSTEM

#1c4966

26 of 44

THE DISPLAY DRIVER

ledDriver.pixel.set({ x: 5, y: 5, color: "#FFFFFF" });

ledDriver.pixel.set([{ x: 1, color: "#ff0000" }, { x: 2 }]);

ledDriver.pixel.setAll(

[

" ",

" # ",

" ",

" # ",

" ",

" "

], "#FFFFFF", true

);

#1c4966

27 of 44

THE DISPLAY DRIVER

ledDriver.image.set("./test.png");

ledDriver.text.scroll("The quick brown fox jumped over the lazy dog");

ledDriver.text.scroll("The quick brown fox jumped over the lazy dog",� "#FF0000", 25);

#1c4966

28 of 44

CREATING A PIXEL ‘FONT’

#1c4966

29 of 44

Convert the pixels to binary values.��filled = 1� empty = 0

CREATING A PIXEL ‘FONT’

[0, 0, 0, 0,

0, 0, 0, 0,

1, 0, 1, 0,

1, 1, 0, 1,

1, 0, 0, 1,

0, 0, 0, 0];

#1c4966

30 of 44

CREATING A PIXEL ‘FONT’

[0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 1, 0, 0, 1, 0, 0, 0, 0];

JIMP to �the rescue

https://www.npmjs.com/package/jimp

#1c4966

31 of 44

CREATING A PIXEL ‘FONT’

#1c4966

Looking at the font image, please imagine this all in one long line, rather than wrapped around like this, I’ve taken liberties for slide design,

You may have noticed, that the characters aren’t all the same width.

I personally think it looks nicer, but it means that the array of bits that we just created will requires a list of indexes which will point to the starting position of each letter and its width.

These two byte arrays are small enough to be embedded on the microcontroller.

With the “font” arrays embedded on the device, it is possible to write code in C/C++ (the language that the microprocessor uses) to show the letters on the display.

This means that instead of computing all of the individual pixel positions in the browser app, and sending them one by one, the microcontroller will handle the translation - saving us precious data during transmission over MQTT.

So, cool, we have a font. Now we need to work out how to display it on the array, which, as always with hardware, isn’t as simple as I’d have hoped.

You see there are multiple different ways that LED arrays can be wired up.

32 of 44

HARDWARE DIFFERENCES

#1c4966

LED displays, like the one used in this project are made up of addressable RGB LEDs.

“Addressable” means that each LED on the display has its own address number, the first LED on the strip is LED 0, the second LED 1, and so on as they move along the matrix.

There is an open source Arduino library, written to interact with these LEDS, called AdaFruit NeoPixel which makes it simple to set the colour value of individual pixels with a pixel address (or ID) and a colour value, set in RGB.

Some common wiring patterns for LED arrays look like these diagrams here

Reading order — with pixel IDs incrementing from left to right.
Snaking horizontally — Pixels increment from left to right, but then reverse on the way back.
Snaking vertically — Pixels increment up and down the matrix.

Not only that, but the displays sometimes have connectors at both ends, which means that the numbering could go from left to right, or right to left, depending on which connector is used.

These differences are mostly made to accommodate physical constraints, snaking the wires uses a lot less wiring than bringing the line back to the start of each row.

This variety of wiring patterns meant that the code running on the board needed to be able to translate from (x,y) coordinates into the appropriate pixel ID, so that developers working with this code will be able to use it with the display they own.

I needed to ensure that all of the operations in the JavaScript refer to pixel locations in (x,y) coordinates rather than pixel IDs.

Once these coordinates are sent to the hardware, it will then translate the coordinates into the correct pixel ID, based on hardware configuration code running on the board.

33 of 44

CONFIGURING

THE ARDUINO

// Display config

const int display_gpio_pin = 4;

const int display_width = 32;

const int display_height = 8;

const index_mode display_connector_location = index_mode::TOP_LEFT;

const carriage_return_mode line_wrap = carriage_return_mode::SNAKED_VERTICALLY;

const neoPixelType neopixel_type = NEO_GRB + NEO_KHZ800;

#1c4966

I can then configure the board to tell it what kind of display it is powering.

The microcontroller code has a single Configuration file included in the project that contains the display configuration, along with configuration for WiFi and MQTT settings:

You can see here that every device needs to be told the GPIO (General Purpose IO) pin the display is connected to, the dimensions of the display,

and the index_mode and carriage_return_mode settings which capture the possible difference between displays.

This means that the code will be able to drive displays bought from different manufacturers, without spending time changing the code.

Working with hardware is such fun!

Ok, so now that we have that under control, lets’ look at how to actually process the messages that the board receives, and display the text.

34 of 44

PROCESSING THE MESSAGES

auto controlCode = (int)bytes[0];

if (controlCode != DeviceControl1) {

Serial.println("Not a device control command.");

return;

}

auto controlMode = bytes[1];

switch (controlMode) {

case P_PixelMode:

processSetPixelsMessage(bytes, length);

return;

case C_ControlMessage:

processControlMessage(bytes, length);

return;

case T_TextMode:

processSetTextMessage(bytes, length);

return;

}

#1c4966

35 of 44

PROCESSING THE MESSAGES

void message_processor::processSetTextMessage(const char *bytes, int length)

{

auto messageModeByte = (int)bytes[2];

auto scrollSpeedMs = (int)bytes[3];

auto r = (int)bytes[4];

auto g = (int)bytes[5];

auto b = (int)bytes[6];

auto shouldScroll = messageModeByte == 1;

drawEntireString(bytes, length, 0, r, g, b);

}

#1c4966

36 of 44

WRITING TEXT TO THE DISPLAY

void message_processor::drawEntireString(const char *bytes, int length, int shiftXposBy, int r, int g, int b)

{

auto textStart = search_utils::indexOf(bytes, length, STX_StartOfText) + 1;

auto footerLength = 2;

auto spaceWidth = 1;

auto xPosition = shiftXposBy;

for (auto i = textStart; i < length - footerLength; i++) {

auto asciiCode = bytes[i];

auto spriteAndWidth = sprites.spriteDataFor(asciiCode);

drawSpriteAtPosition(spriteAndWidth.spriteData, spriteAndWidth.width, xPosition, r, g, b);

xPosition += spriteAndWidth.width + spaceWidth;

}

lights_->flush();

}

#1c4966

When a message to display text is received, the code loops through each text character in the message and looks up that character in the embedded font.

The font data that gets returned from the font is the array of bytes we saw earlier, prefixed by the width of the character in pixels.

Because the font data includes the width, when the code iterates over subsequent characters, it can calculate the location it needs to start drawing at.

This location is called xPosition in the code, and each time a character is drawn on the display, its width and an additional space is added to the xPosition.

As the code loops, all of the (x,y) coordinates returned from the font have this xPosition added to them making sure that the characters are drawn in sequence.

This continues until the loop reaches the width of the display, where it stops drawing the extra characters that would flow off the side.

So that will write text to the display, but we also want to scroll it.

37 of 44

SCROLLING

THE TEXT

ON THE

DISPLAY

#1c4966

38 of 44

SCROLLING THE TEXT

if (shouldScroll) {

auto maxWidthOffLeft = sprites.getTotalWidthOfText(bytes, length) * -1;

auto scrollPosition = cfg_->display.width;

while (scrollPosition > maxWidthOffLeft) {

lights_->clearWithoutFlush();

drawEntireString(bytes, length, scrollPosition, r, g, b);

scrollPosition--;

delay(scrollSpeedMs);

}

} else {

drawEntireString(bytes, length, 0, r, g, b);

}

#1c4966

39 of 44

SCROLLING THE TEXT

if (shouldScroll) {

auto maxWidthOffLeft = sprites.getTotalWidthOfText(bytes, length) * -1;

auto scrollPosition = cfg_->display.width;

while (scrollPosition > maxWidthOffLeft) {

lights_->clearWithoutFlush();

drawEntireString(bytes, length, scrollPosition, r, g, b);

scrollPosition--;

delay(scrollSpeedMs);

}

} else {

drawEntireString(bytes, length, 0, r, g, b);

}

#1c4966

40 of 44

IT WORKS!

#1c4966

41 of 44

NO DISPLAY?

NO PROBLEM!

https://live-caption.ably.dev/

#1c4966

42 of 44

NO DISPLAY?

NO PROBLEM!

#1c4966

43 of 44

MAKE IT YOUR OWN

github.com/ably-labs/live-caption-demo

#1c4966

44 of 44

THANK

YOU

FOR

LISTENING

Jo Franchetti

DevRel at Ably

@thisisjofrank

@ablyrealtime

#1c4966