1 of 72

WP7 Increasing the senses, increasing inclusion

Sonification Workshop - July, 2022

​

Johanna Casado ( ITeDA, UM)

Wanda Díaz-Merced (EGO)

Gonzalo de la Vega (ITeDA)

Pierre Chanial (APC, EGO)

Natasha Bertaina (UM)

Gary Hemming (EGO)

B García (ITeDA, UTN)

……………

​

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

2 of 72

Introduction

Part 1.A

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

3 of 72

Astronomy (and Sciences in general) as a professional field, only benefits those that can cope with the performance styles currently available.

​

Anyone may develop a disability and currently this population has no way to perform a real contribution to the Science.

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

4 of 72

Almost all that we know about the Universe is not in the visible region

​

The Astrophysics multi-wavelength and multimessenger

opened a new window to the Cosmos

5 of 72

Accessibility: tactile resources

6 of 72

7 of 72

8 of 72

Original Developments:

​

  • Electronics Design.
  • Software design.
  • Scripts and music.
  • Models.
  • 3D prints and Braille.
  • Sign language videos.
  • Documentation.
  • Repositories.

9 of 72

  • Use of resources and techniques tested

​

  • Evaluation of new resources, ideas and revision: feedback with users

​

  • Blind and deaf astronomers as advisors.

​

  • Study of Impact - different audiences.

Considerations for the new approach:

SHOW

10 of 72

Participation in the regular stream of performance has to be taken into account from the beginning in the development agenda

​

Big Data drives all inclusive aspects of social and human development.��Regardless, it appears that international efforts have been limited entirely to developing packages that address big data visualization.

11 of 72

Inclusion and Diversity

Sonification Platform and Training Courses

A new user-centered software to produce audio-visual outputs from 1d and 2D data; it integrates multidisciplinary and interdisciplinary disciplines and scientists, both blind and sighted, and addresses the topics of accessibility to scientific data

​

Platform for Artistic Intervention

A platform to support the cross-reflection between artists and scientists, of the field of fundamental science.

​

​

​

​

21 September, 2021

Open Science Fair- Increasing the senses, increasing inclusion

12 of 72

A preliminary study showed that some of the programs available to sonify large data sets and symmetrically display the graph are not accessible according to ISO 9241-171: 2008.

(Guidance on software accessibility)

We present a User Centred approach to develop data analysis and data retrieval tools that will permit people with other sensory styles to explore scientific data and make science

(Based on theoretical frameworks established from attention mechanism and coping strategies of people with visual disabilities and focus group analysis)

13 of 72

OVERVIEW of the proposal

13

Strategy

Implemen tation

Evaluation

Objectives

Challenges

Design-task

Comm. Public

Use-Feedback

Surveys

Development

Deliverables

Training lessons

14 of 72

sonoUno bases

  • Since 1992 International Community for Auditory Displays (ICAD) gather publications about display and perception of sonification to research and outreach.
  • The use of multi-sensorial exploration of data not only makes it accessible, also enhance the actual display.
  • Since 2017 sonoUno is being developed actively; its web interface, makes possible data exploration directly from the web and can be used with screen readers.

15 of 72

Motivation

From the beginning we search the way to make an inclusive tool, we actively work to make an adaptive software for different functional diversity, allowing total autonomy; so the people can explore, research and talk with colleagues as equals.

16 of 72

This proposal integrates multidisciplinary and interdisciplinary astronomers, engineers, software designers, educators, disability specialists, neurologist, sociologists, both able and disabled.

​

Jesse Leaman

Wanda Díaz-Merced

Johanna Casado

Aszia Baicouchi

Jeffrey Cooke

Garry Foran

Gonzalo de la Vega

17 of 72

  • link the people with a user centred development.

​

  • All people must have the same rights and all people must be able to make science!

​

18 of 72

The sonorization

Part 1.B

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

19 of 72

A new software design, a new approach to the data

1. Accessibility to scientific data, from the Earth or with instruments on board satellites (available in databases).

�2. Creation of a human-computer interface suitable for the access, collection, sonification and analysis of astrophysical data.

​

3. Test the efficiency, usefulness and effectiveness of the resource in different cultural settings.

​

4. Develop the paradigm for training researchers and interested citizen scientists to start using new techniques.

20 of 72

sonoUno main characteristics

  • Use of sound as a complement to visualization.

​

  • A user-centered design from the start allows to combine accessible features with the necessary scientific efficiency.

​

  • Completely open source, cross-platform and multiple device.

​

  • Aims to eliminate the barriers presented by current technologies for people with sensory and motor disabilities.

​

  • Allows to improve the work with different styles of data exploration.

21 of 72

OUTPUT

  • Sound
  • Plot
  • Plotted data
  • Marked points

INPUT

  • Two column file (csv or txt)
  • Sound (in progress)
  • H5 data
  • FITS

SonoUno DISPLAY

  • Visual plot
  • Sound of the plot
  • Marks on the data
  • Grid with the data
  • All functionalities

22 of 72

Modular design

Core Module

Data Export

Data Import

Data Transform

GUI

Sound Module

23 of 72

SonoUno allows for the input of large datasets

Input of data with a million of rows has been tested.

24 of 72

Presets

Synthesis

Harmonics

Envelope

Wave form

File

Device mixer

SonoUno sound synthesis module

Wave synthesis allows for arbitrary frequencies and timing, and also altering the sound at will.

25 of 72

sonoUno

Part 1: Functionalites

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

26 of 72

sonoUno

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

27 of 72

Highlights

​

  • - Platform and Tools for Visually impaired citizen scientists (desktop & web)

​

web-interface

  • sonoUno web-interface
  • sonoUno general website

sonoUno: A USER CENTERED DESIGN

28 of 72

Highlights

SonoUno webpage

29 of 72

Highlights

SonoUno web page

  • short-cuts
  • collapsible panels, following�sonoUno framework

30 of 72

Highlights

SonoUno web page

  • Menu with I/O elements

  • Sample data:
  • REINFORCE demonstrators
  • Simple functions

31 of 72

Highlights

  • Editable names

​

  • Sound presets

​

32 of 72

Highlights

33 of 72

sonoUno

Part 2: Detection

Handling of sample data with guidance

(Sequence of activities) (15-20 minutes)

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

34 of 72

sonoUno

Part 2: Detection multiple devices

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

35 of 72

The LightSound

36 of 72

Highlights

​

  • Critical and Scientific Thinking: The Big Ideas of Science

36

Solar Eclipses

LightSound (Harvard)

Total Eclipse

LightSound

SUNLIGHT

SOUND

DATA

Partial Eclipse

LightSound

Partial Eclipse

sonoUno web!!

More than 100,000 citizens

successful story

37 of 72

38 of 72

SONOUNO: evidence of use in Education

Carlos Morales Socorro with students.

Collaborators: Alejandro Bolaños, Moisés García , Floro Robaina, Rosetta Martorell, Francisco Alvarado y Ángel Marrero. (Asociación Astronómica y Educativa de Canarias "Henrietta Swan Leavitt", Gran Canaria, Spain)

​

​

The tool is used for training young students in the study of variable stars.

39 of 72

SONOUNO: evidence of use in Education

40 of 72

SONOUNO: discovery of a variable star

CEP Las Palmas, May 17, 2022, the exact moment in which the students became aware that they were the first students in the history of Humanity to discover a variable star through the sonification of signals and the use of tactile diagrams.

More info

41 of 72

SONOUNO: evidence of use with real data

Pierre Auger Open Data

42 of 72

The exchanges

with the users determined the engineering of the algorithm leading to a modular design and the deployment that the sonoUno has today.

  • [...] I have more confidence and freedom as well, I felt like I was a lot more able to access the bitly data that I want to do.

​

​

  • [...] It was very easy to change the way things sounded, easy to choose something that I like the sound of that works for me. [...]
  • [...] the circumstances where you are working with people that have, literally, no vision, it's quite valuable because it's give you access to something that you otherwise wouldn't have access to [...]

​

43 of 72

sonoUno

Part 3: REINFORCE demonstrators

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

44 of 72

Demonstrators

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

45 of 72

Training

​

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

46 of 72

Training activity overview

DAY 1

DAY 2

REINFORCE data demonstrators

    • GWhitchHunters: training
    • New particles at LHC: training
    • Cosmic Muons images: training

​

  • detection/understanding
  • detection/discovery

​

47 of 72

Challenges for REINFORCE

  • Sonorize the data from ALL the demonstrators
  • Produce an unified version
  • Give visibility to the proposal�website�training lessons, perception analysis

successful stories

  • Assure communication with the public�citizen scientist as part of the development�exhibitions, new materials�webinars, virtual visits (inclusives)�hands-on activities

​

48 of 72

Highlights

SonoUno webpage

  • Sonification and user centered design bring access to astrophysics, including people with functional diversity.

49 of 72

GALLERY UPDATES (to link in REINFORCE website)

https://www.sonouno.org.ar/reinforce-demonstrators/

50 of 72

GALLERY UPDATES

51 of 72

SONOUNO FOR DIFFERENT DATA SETS

www.sonouno.org.ar

https://github.com/sonoUnoTeam/sonoUno

52 of 72

REINFORCE Scripts

    • python 3.x installed
    • The folder sound_module (from github) or the full soft
    • During installation in Windows, you must "set the environment path" by clicking in the box that appears on the screen.
    • pip is installed
    • Install the libraries with pip:
      • python3 -m pip install matplotlib numpy pandas pygame
      • For Windows: python -m pip install matplotlib numpy pandas pygame opencv-pythonv

53 of 72

IMAGE sonorization: glitch at Virgo detector

​

  • python3 img_sonif.py -d "path_to_the_image_to_sonify"

​

54 of 72

SONIFICATION OF LHC DATA SETs

  • an electron - represented by a track in the inner detector that points to a cluster in the calorimeter
  • a converted photon - represented by two very close tracks in the inner detector, that points to a cluster in the calorimeter;
  • a muon - represented by a long track that goes through all of the detector layers and which could, although this is not necessarily the case, be a point to a cluster;
  • a photon - represented by a cluster in the calorimeter, but with no track in the inner detector;
  • or an unknown particle- any other representation that is not covered by any of those above.

​

cross-section

lateral view

55 of 72

SONIFICATION OF LHC DATA SETs

​

  • python3 lhc_bash.py -d "path_to_the_lhc_data_file"

​

​

​

  • python3 lhc__display_bash.py -d "path_to_the_lhc_data_file"

​

​

  • For a given data set and produce the images and sounds in the same location as the data file
  • The script opens a new plot window where the tracks will be displayed during the reproduction of the sound. After the reproduction, the sound (wav) and images (png) are stored in the same folder as the data file

56 of 72

SONIFICATION OF LHC DATA SETs

cross-section

lateral view

  1. center of the detector and the beginning of each particle track is indicated by a beep .

​

  • Track or non rack in the inner detector::
    1. Continuous sound (piano note ‘D6’, frequency 1174.66Hz), 2s or 4s if the particle is a muon.
    2. If two tracks, two continuous sounds at two different frequencies: piano notes ‘D6’, 1174,66Hz; and ‘C6’, 1046.50Hz. ( 2s).
    3. In the case of non track, a silence of 2s.

​

  • End of the inner detector, beginning of the calorimeter: a second tick mark, a sound of a short ‘F7’ piano note, of 2793.82Hz (1ms).

​

In the case of muons, this tick mark is sonified after 2 seconds of the beginning of the continuous sound (the muon passes from the inner detector to the calorimeter)

).

​

Clusters in the calorimeter: a specific compilation of short sounds; volume = energy.

​

I

57 of 72

SONIFICATION OF LHC DATA SETs

Plot for Event 326146241.

​

On the left, a transversal view, and on the right, a longitudinal view, of the full detector.

​

The clusters are represented by black circles.

​

The sonification action-log is displayed in the area at the bottom.

​

58 of 72

COSMIC MUON DATA SONIFICATION

graphical representation of the Cosmic Muon Image data, showing the three layers of the detector

example of the presence of a muon in the data

example of NON presence of a muon in the data

59 of 72

COSMIC MUON DATA SONIFICATION

  • Each of the 32 or 16 channels of each plot on the right/left was assigned to specific 16 piano notes.

In the case of more than one channel presenting a hole a sound is a combination of notes

This is a work in progress

$ python3 muon_bash.py -t "csv" -d "Path of the image" -p True

60 of 72

Training activity display

A. Signal shown to the participant, accompanied by the audio signal.

B. Options offered to the participant, who must use the keys to respond as they see fit.

61 of 72

DAY 2

​

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

62 of 72

Sonification Training Activity Results

62

63 of 72

Sonification Training Activity Results

63

64 of 72

Conclusions

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859

65 of 72

Sonification-as-a-service platform

Researchers, artists, citizens craft a transform

Transform

Transform ID

sonoUno server stores the transform and returns a unique identifier

66 of 72

Transforms : Directed Acyclic Graphs

  • An audio transform is described by a DAG, that could be similar to a classical data reduction pipeline:
      • denoising
      • whitening
      • deglitching
      • …
  • Several data channels can be used as input
  • The output can be mono, stereo or binaural audio
  • Each transform node can execute a Python (and in the future MatLab) script

67 of 72

Sonification-as-a-service platform

Transform ID + data

audio file

sonoUno server executes the transform on the data

A sonoUno client needs a sonification.

​

Broader audience: zooniverse, ESFRIs

68 of 72

Challenges

Architecture for a RESTful API server enabling connection of REINFORCE Zooniverse demonstrator projects to sonoUno.

​

69 of 72

Challenges

The future: adopting the technique

  • Suggest nations to carry out assessments about the gap between what organisations report as achieved and the quality of participation of the disabled.

​

  • Systematise a report format where to have evidence of UCD in the development of prototypes and databases.

​

  • Recognise multi-sensorial exploration as valid study of the data.

​

  • Stimulate funding agencies to include in the projects people with disabilities and other diversities.

70 of 72

Challenges

The future of sonificationts

  • Increase the abilities to identify signatures in the information. 

​

  • Bring peoples with disabilities to the field.

​

  • Increase the amount of scientific discoveries.

Open Science accepting different data exploration styles, more perspectives and experiences.

71 of 72

Conclusion

The cause of the inequity was addressed.

​

The systemic barrier has been removed.

Courtesy Advancing Equity and Inclusion: A guide for municipalities, City for All Women Initiative (CAWI), Ottawa

INCLUSION!

72 of 72

Join the community

Reinforceeu.eu

/company/reinforceeu

@ReinforceEU

© Copyright 2019 – This project has received funding from the European Union’s Horizon 2020 project call

H2020-SwafS-2018-2020 funded project Grant Agreement no. 872859