Harvard-MIT Speech and Language Biomarkers Interest Group

Join or invite others to join mailing list: https://groups.google.com/forum/#!forum/harvard-mit-speech-group/join

Daniel M. Low, MSc, PhD student, Harvard University and MIT

1st and 3rd Monday of each month. Time: 11am ET (Boston time)

Daryush Mehta, PhD, Associate Professor of Surgery, Department of Surgery, Harvard Medical School

90 minutes (40 min presentation + 20-40 min of Q&A, but you can come for less time if you prefer)

zoom: https://harvard.zoom.us/j/92586896164?pwd=0DuG5aSDvgVWpzOdqp5Tupl95F8sHo.1

Hamzeh Ghasemzadeh, PhD, postdoc, MGH

Tanya Talkar, PhD, Aural Analytics

Satrajit Ghosh, PhD, Principal Research Scientist, MIT McGovern Institute for Brain Research

Thomas Quatieri, PhD, Senior Technical Staff member, MIT Lincoln Laboratory

Rahul Britto, PhD Student, Harvard & MIT

Provide ideas / speaker suggestions by sending any of the organizers an email

Nick Cummins, PhD, Kings College London

Unsubscribe: To stop getting emails from this group, you can send an email here (with no subject or body text) using the email with which you subscribed:
harvard-mit-speech-group+unsubscribe@googlegroups.com

Prior slides, papers, and code here

20-Jan-2025 (Martin Luther King Jr. Day)

"Clinically meaningful speech-based endpoints in clinical trials"

Traditional clinical rating scales used to assess neurodegenerative motor conditions are often too coarse to detect subtle changes in disease progression, contributing to high failure rates in clinical trials. While these scales are valuable for broad assessments, they lack the sensitivity needed to capture nuanced changes in motor function over short intervals. In contrast, speech offers a promising alternative as an endpoint because it is both a core symptom of many conditions and can be measured frequently and remotely, enabling precise tracking of progression with minimal patient burden.

This talk demonstrates the successful application of speech as an endpoint in amyotrophic lateral sclerosis (ALS) clinical research. We begin by reviewing the pathophysiology of ALS and the impact of motor neuron degeneration on the speech mechanism. Next, we explain why speech serves as a functional and meaningful indicator of bulbar impairment and disease progression. Finally, we highlight the advantages of speech-based endpoints over standard scales, illustrated by a clinical trial in which pre-specified analyses using speech metrics revealed a significant slowing of disease progression.

17-Feb-2025 (Presidents’ Day)

"Exploring Intraspeaker Variability in Vocal Hyperfunction Through Spatiotemporal Indices of RFF"

Abstract: Hyperfunctional voice disorders are highly prevalent yet difficult to objectively characterize. Relative fundamental frequency (RFF) has potential for characterizing these disorders but faces limited clinical use due to intraspeaker variability in mean RFF values. However, what if this variability contains valuable diagnostic information? Instead of minimizing or disregarding variability, could it be leveraged to enhance RFF's discriminative power in assessing vocal hyperfunction (VH)? To explore this idea, we examined two measures of variability: a basic measure, standard deviation (SD), and a more complex metric, the spatiotemporal index (STI). As RFF is a measure of how instantaneous fundamental frequency changes as someone stops or starts voicing, it is interestingly parallel to STI, which assesses movement pattern stability exhibited over repeated performance of the same motor task (such as speech production). By applying SD or STI to RFF estimates, we aimed to determine whether incorporating measures of variability can enhance our ability to discriminate between VH subtypes and individuals with typical voices, potentially capturing aspects of the condition that mean RFF values alone may miss." Jenny Vojtech , BU

"Speech as a Biomarker for Disease Detection"
Abstract: Today’s overburdened health systems face numerous challenges, exacerbated by an aging population. Speech emerges as a rich, and ubiquitous biomarker with strong potential for the development of low-cost, widespread, remote testing tools for several diseases. In fact, speech encodes information about a plethora of diseases, which go beyond the so-called speech and language disorders, and include neurodegenerative, psychiatric, and respiratory diseases. Recent advances in speech processing and machine learning have enabled the automatic detection of these diseases from speech. Despite promising results, this research area faces challenges, primarily due to dataset limitations and the overlap of speech-affecting diseases, which often coexist and produce similar speech manifestations. These challenges guide our latest research, where we discuss the characterization of normative speech. Similar to common blood tests, we explore reference intervals for interpretable speech features (acoustic and linguistic) as a first step toward adopting speech analysis for multidisease screening. We leverage deviations from these references to detect Alzheimer’s and Parkinson’s diseases using different classifiers, namely Neural Additive Models for enhanced interpretability. Additionally, we explore bridging black-box models and interpretability by using large language models to annotate high-level, low-dimensional, interpretable speech characteristics, termed macro-descriptors—such as text coherence and lexical diversity. Using only four macro-descriptors, we outperformed conventional language-based Alzheimer’s detection methods. This talk aims to deepen our understanding of speech’s multifaceted potential as a biomarker for holistic health.

Dr Catarina Botelho
Bio: Catarina Botelho received her B.Sc. and M.Sc. degrees in Biomedical Engineering from Instituto Superior Técnico (IST), University of Lisbon, in 2018, and completed her Ph.D. in Electrical and Computer Engineering also from IST in 2024. Her research, conducted at INESC-ID, focuses on speech as a biomarker for disease detection, particularly in conditions such as obstructive sleep apnea, Parkinson’s Disease, Alzheimer’s Disease, and COVID-19. Her MSc and PhD work has been distinguished by two awards from IST and the University of Lisbon. Currently, she is a researcher at INESC-ID, contributing to the Accelerat.AI project. She has held positions as a research intern at Google AI, Toronto, and as a visiting researcher at the Cognitive Systems Lab, University of Bremen. She was involved in the student advisory committee of the International Speech Communication Association (ISCA-SAC), since 2020 to 2023, acting as Coordinator in 2022. Her scientific interests lie on speech and language technology for healthcare, and their combination with other modalities.

Exploring the Mechanistic Role of Cognition in the Relationship between Major Depressive Disorder and Acoustic Features of Speech

Toward generalizable machine learning models in speech, language, and hearing sciences: Estimating sample size and reducing overfitting

Hamzeh Ghasemzade, PHD (Research fellow, Massachusetts General Hospital – Harvard Medical School)

Clinical theory and dimensions of speech markers: Psychosis as a case study

The use of speech-derived markers in mental health offers a promising avenue for targeted prevention in early intervention. However, we face a problem of plenty: the sheer number of speech markers present significant challenges for clinical translation. Successful real-world predictions require the underlying model to be a ‘reasonable facsimile’ of the clinical phenomena that map on the predicted outcomes. In search of such theoretical mapping, most of the research to date have focussed on cognitive basis of language. Here I will make a case for psychopathology of illness phenomena to provide the theoretical framework for dimension reduction. Using the example of psychotic disorders - especially schizophrenia, I will argue that the psychopathological theory-based speech-marker models provide incremental value to the clinical intuition. I see opportunities for this to be applied in the study of depression, bipolar disorder and substance intoxication.

Lena Palaniyappan, PhD (Professor of Psychiatry, McGill) https://www.mcgill.ca/psychiatry/lena-palaniyappan

Dr. Lena Palaniyappan is a practicing psychiatrist; he works with youth and families experiencing severe mental illnesses such as psychosis. Following his Bachelor’s degree in psychology, he completed his medical training at Stanley Medical College in Chennai, India followed by a Master’s and PhD in Psychiatry at the University of Nottingham in the UK. He currently holds the Monique H. Bourgeois Research Chair and directs the Centre of Excellence in Youth Mental Health at the Douglas Research Centre. He is also the Chief Editor of the Canadian College of Neuropsychopharmacology Journal. His work on neuroimaging in psychosis led to the Global Rising Star Award from the Schizophrenia International Research Society (SIRS) and a Canadian Institutes of Health Research (CIHR) Early Career Foundation Grant. His research program is geared towards optimizing long-term mental health outcomes and pathways of care for individuals with serious mental disorders that often start in adolescence. His work centers on developing an understanding of the brain mechanisms involved in mental states such as psychosis and depression and in generating developmentally informed tools to predict outcome after first-episode psychosis and non-invasive treatment approaches.

Maria Powell, PhD (Vanderbilt University Medical Center, Department of Otolaryngology-Head & Neck Surgery)

Dr. Powell, is a Research Assistant Professor in the Department of Otolaryngology at Vanderbilt University Medical Center. She is an ASHA-certified speech-language pathologist, with expertise in the assessment and management of voice disorders. Her work bridges clinical research and artificial intelligence by building collaborative frameworks that integrate the voices of diverse research partners—including patients, clinicians, and engineers—into the design, development, and evaluation of AI systems. She is particularly passionate about responsible data practices, inclusive research design, and advancing equitable outcomes in healthcare technology.

Roseline Polle (Thymia)

Roseline is a Machine Learning Researcher at Thymia. She works on subjects such as AI quality, confounding variables and longitudinal modelling.
She has experience in AI risk management, having conducted bespoke audits for AI systems across various sectors at Holistic AI. In addition to auditing, she has contributed to developing AI governance, risk and compliance platforms. Before that, she worked in the Energy sector as a computational analyst. Her educational background is in Machine Learning and Engineering.

Ingo Siegert, PhD (Otto-von-Guericke-University Magdeburg) https://www.ingo-siegert.de/

Since November 2018, Jun.-Prof. Ingo Siegert has been leading the Group of Mobile Dialogue Systems at Otto-von-Guericke University Magdeburg. Following his studies in Information Technology and an internship at IBM Germany, he completed his Ph.D. in 2015 with a focus on automatic emotion recognition in spoken language. In his interdisciplinary research, he explores natural human-machine interaction, investigating both human speech behavior towards machines and human feedback signals. Additionally, he researches methods for emotion-preserving speaker anonymization and strategies for enhancing trust in assistive technology.
Currently, Ingo Siegert supervises 4 doctoral students in 5 research projects (3 of which he leads as part of a consortium). He has published over 100 peer-reviewed contributions in conference proceedings and journals and has played an active role as an organizer in numerous workshops and conferences. He currently serves as the Head of the ITG Department for Services and Applications and as the Secretary of the ISCA Special Interest Group "Security and Privacy in Speech Communication." Furthermore, he is involved in various formats of science communication and student recruitment.

Fan Wu

Fan Wu is a Ph.D. candidate at ETH Zurich, Switzerland. She holds a master’s degree in electrical engineering and information technology from TU Munich, Germany. Her research aims to advance personalized healthcare using innovative digital tools. She focuses on developing biomarkers for various conditions, such as acoustic biomarkers for heart failure. Her role includes managing research studies end-to-end, including study design, mobile app and web server development, data collection and data analysis.

Godino-Llorente, PhD https://scholar.google.com/citations?user=fdkx_u4AAAAJ&hl=es

Modelling individual and cross-cultural variation in the mapping of emotions to speech prosody

Abstract: The mapping between emotions and speech prosody is commonly studied by training classifiers on acoustic features. In this talk, I will present our recent Nature Human Behaviour paper in which we propose a Bayesian modeling framework to analyze this mapping. In the paper, we fit our models to a large collection of intended emotional prosody from across the globe. Our descriptive study reveals that the mapping within corpora is relatively constant but varies across corpora. We fit a series of increasingly complex models to account for this heterogeneity. The model comparison reveals that models considering mapping differences across countries, languages, sexes, and individuals outperform models that only assume a global mapping. Further analysis shows that differences across individuals, cultures, and sexes contribute more to the model prediction than a shared global mapping. In the talk, I'll highlight some of the limitations of the study (biased sampling of emotional prosody, cross-lingual alignment of emotion terms, heterogeneity in recordings) and outline how human-in-the-loop experiments can solve these issues.

Pol van Rijn. PhD student in the Research Group Computational Auditory Perception at the Max Planck Institute for Empirical Aesthetics. He developed various human-in-the-loop algorithms to study semantic representations. He applies them in massive online experiments across many languages and countries to overcome some core problems in emotion science (biased sampling, cross-lingual alignment, lack of control in corpus analysis). The developed cognitive pipelines can also be used to study machine representations and have applications in various fields, including voice design and robotics. By combining methods from computer and cognitive science, he aims to address long-standing problems in emotion science by using novel approaches.

Mohammad Soleymani, PhD (USC). Research Associate Professor in Computer Science at USC Institute for Creative Technologies and USC Viterbi School of Engineering and the director of the Intelligent Human Perception Lab.

Reading group session: Introductory overview of self-supervised learning, transformers, and attention. Please read or skim through the following blogs and papers:
- https://jalammar.github.io/illustrated-transformer/
- https://sebastianraschka.com/blog/2023/self-attention-from-scratch.html
- Liu ... & Schuller, B. W. (2022). Audio self-supervised learning: A survey. Patterns.