1 of 5

VISUALIZING OCEAN SOUNDSCAPES �IN VIRTUAL REALITY

TONGYU ZHOU

2 of 5

MOTIVATION

  • Sound can convey vital information about environments in instances where it is difficult to obtain visual data
  • Studying ocean soundscapes allow us to better understand the relationships between biophony (sounds of the marine life), geophony (sounds of the earth), and anthrophony (sounds of human activities)
  • Goal:

Enable immersive collaborative viewing and annotation of wave fragments

Audio data

3D spectrograms projected to virtual plane in VR

Image source: https://www.youtube.com/watch?v=SnmryX5cxr4

Humpback whale

3 of 5

DATASET

​

  • Audio recordings from the Monterey Bay Aquarium Research Institute (MBARI), captured using MARS hydrophone, maintained since 2015
  • Samples >250,000 times each second to capture broad audio ranges (256 kHz)

Image source: https://www.mbari.org/mars-hydrophone/

4 of 5

MILESTONES

  • 2/15: Obtained a copy of the audio recordings. Reviewed audio processing software (ObserVIEW, Insight, SignalScope) to produce spectrograms (to check later Unity implementation) 🡪 3D spectrograms are all paid… 🡪 ended up plotting the spectrograms in Python

Deliverable: Images of the spectrograms.

12/31/2021 – first 15 minutes, sampled at 16 kHz

Selected dolphin sound snippet – 30 seconds, sampled at 32 kHz

5 of 5

MILESTONES (CONT.)

  • 2/17:  Started a new project in UnityXR + finished HelloWorld-esque tutorials. Imported audio data into Unity and listen to it in VR.

Deliverable: Video of listening to audio recordings in VR.

​

  • 2/22 (2/24 is a holiday):  (Unity) Visualized audio data as spectrograms. 

Deliverable: Video showcasing this in action. A tutorial in the VR software wiki going over signal processing/visualization with Unity.

​

  • 3/01: Implemented functionality to allow users to annotate and control both audio playing and the corresponding spectrogram animation.

Deliverable: Video recordings in VR.

​

  • 3/03: Researched existing plugins for multiplayer VR in Unity.

Deliverable: New plugin entries and additions to existing plugin entries are added to the “Multiplayer VR Development in Unity” page.

​

  • 3/08: Spectrogram interactions are now collaborative. 2 people can enter the same visualization and annotate the sound effects simultaneously. A class activity where users are instructed to use the system to add 3 annotations for sounds to the corresponding wave snippet.

Deliverable: the in-class activity

​

  • 3/10: Based on the results of the previous activity, a completed entry in the wiki describing the exploration of collaborative sound annotation in VR.

Deliverable: An entry for “Soundscape Ecology” in the “Applications of VR” page with takeaways.�