1 of 18

ChatAssist

Ron, Laurissa, James, Andrew

2 of 18

What we set out to do

what’s up Mark

that’s on the graph

View Full Log

that’s on the graph

​

what’s up Mark

3 of 18

MVP

Converse with one person and see what they say in a chat bubble

Specific Features:

  • Audio is recorded (it may or may not be sent to MagicLeap, depending on how we end up implementing it)
  • Audio is sent to a Speech-to-Text API
  • Response from API is sent to MagicLeap
  • Speech bubble is displayed to the user with speech-to-text response at some fixed distance from user

​

​

4 of 18

MVP

Converse with one person and see what they say in a chat bubble

Specific Features:

  • Audio is recorded (it may or may not be sent to MagicLeap, depending on how we end up implementing it) ✔
  • Audio is sent to a Speech-to-Text API ✔
  • Response from API is sent to MagicLeap ✔
  • Speech bubble is displayed to the user with speech-to-text response at some fixed distance from user ✔ and with basic tracking

​

​

5 of 18

Challenges

6 of 18

Image Tracking

Tracking is not ideal

  • Limited to 1.5m �(Magic Leap example docs)
  • Can’t take videos
  • Tracking is jittery
  • Weird Unity bugs

7 of 18

Speech-to-Text APIs

  • API compatibility issues
    • Double trouble w/ Unity and Magic Leap�
  • Workaround: servers
    • WebSockets for client-server connection

8 of 18

Microphones

  • Microphones don’t work all the time
  • Not an issue specific to this class
  • Still haven’t gotten Bluetooth mic to work
  • Will get better mics

9 of 18

Current Design

10 of 18

Language Selection

*

*

*

Means it turns transparent

11 of 18

Chat Bubbles

12 of 18

Fun Infrastructure Stuff

13 of 18

Fun Infrastructure Stuff (part 2)

14 of 18

Fun Infrastructure Stuff (part 3)

​

​

15 of 18

Demo Video

16 of 18

Demo Video

Note: We wanted to include a video off the Magic Leap, but image tracking requires the use of the cameras, so we aren’t able to take a video while our app runs.

17 of 18

What’s next?

  • UI and tracking improvements
    • Change language selection to be mounted on controller
  • Multi-speaker support
    • Requires tracking multiple people at once and multiple server requests
  • Better controls
    • Laggy, but probably better once we port it.
  • Speech-to-text performance fixes
  • Transcript logging and display
    • Support is present, but we haven’t tested it yet
  • Automatic translation; select speaker and display language

​

Planning; Started; Almost Done; Needs Testing

18 of 18

Questions?