InSync
B4: Aakash Sell, Ben Martinez, Mathias Thomas
18-500 Capstone Design, Fall 2024
Electrical and Computer Engineering Department
Carnegie Mellon University
System Architecture
Product Pitch
One of many challenges of live musical performance is maintaining synchronization between performers. Especially for new musicians, it can be difficult to quantify at what point they have become desynchronized with high precision.
InSync is a rehearsal tool that assists new musicians in pinpointing the precise moments of desynchronization and quantifying the margin of separation between them. After uploading their own sheet music, musicians will receive post-performance feedback, marking gaps of separation greater than 15mS between each other and the sheet music template respectively. This data can be stored, allowing for users to view their improvement over time.
For music pieces with intermediate complexity, our system correctly detects and identifies note start timing within 15 milliseconds for 80% of all notes played. It correctly assesses pitch, each note out of 128 values with a chance value of 1/128, 89% for the recorded vocals and 77% for piano audio. Delays between musicians over 15 milliseconds are reported at the correct start time with 90% accuracy and with the correct duration, within 15 milliseconds, 20% of the time.
The system records audio into Scarlett audio interfaces from two microphones: A cardioid condenser for the singer, and a stereo pair (ORTF). Users upload the sheet music which is broken down into a list of pitch, onset time, and duration data with the help of the Audiveris library. Using Dynamic Time Warping, audio data is compared to the sheet music uploaded by the users. A copy of the music is highlighted where the program detected the two musicians were out of sync.
https://course.ece.cmu.edu/~ece500/projects/f24-teamb4/
System Description
System Evaluation
Conclusions & Partners
The architecture of this project can be broken down into three major components, the web application, the timing algorithm and the audio processing. Figure 1 shows the high level relationship between the components which can be broken down into the following steps: the web application sends parsed sheet music over to the timing algorithm, the audio processor sends processed audio information to the timing algorithm, timing algorithm sends comparisons to web application.
Audio Processing is a complex process that requires extensive testing and optimization to get correct. Though our initial results are promising the effectiveness of our system drastically decreases with more complex pieces of music.
Large portions of our valuable live audio data were collected in collaboration with Dr. Dueck and School of Music musicians.
Live Testing Results:
Piano Mic
Singer Mic
Web App
Raspberry Pi
Audio Interfaces