Cog*ify
BWSI Cog*Works 2023
By: Aadi Shah, Rohan Dugad, Medha Mittal, Khoi Nguyen, Aswin Surya, and Pyae Sone Hmine (TA)
Problem
Which do you choose?
Maintain excitement with high energy music
OR
Spend more time with your friends
No DJ? No problem!
01
Deployment pipeline
03
Training pipeline
04
Music Video Generation
05
Future improvements
02
Demo
The creation of Cog*Ify
Good ol’ machine learning
Diffusion models!
Cog*Ify 2.0?
Cog*Ify in action
Deployment pipeline
How does the program work?
Graphical user interface
Input the Spotify playlist link
Create the Mashup
Query Spotify Playlist
Song order is updated
Getting (NCS) Song data
Spotify API
Youtube API
Scraping song data
ClippinG Songs
Use Spotify’s built in API to get the song names and artists in a playlist
Use Youtube’s search API to get videos of the songs from the spotify playlist
Scrape the song metadata from YouTube
Send the data to the model to get timestamps and then clip the songs
Generate spectrograms
Model prediction
Model prediction
Model prediction
Spectrograms
20000 -
17500 -
15000 -
12500 -
10000 -
7500 -
5000 -
2500 -
f [Hz]
0 -
100 200 300 400 500
t [sec]
Cosine similarity
Determining the best order
First song
NEXT SONG
REPEAT
Order complete
Pick a random song to play first using the random module.
Find which song has the highest cosine similarity to the first song.
For all songs in playlist, use cosine similarity to find next unplayed song.
The order has been determined. . . next transitions!
Adding transitions between songs
Fade in
crossfade
Fade out
Apply a 4 second fade in to the first song in the mashup.
Apply 2 second crossfades between songs.
Apply a 4 second fade out to the last song in the mashup.
Final product
Check how far you are in the mashup!
Skip to other parts of the mashup!
Pause and unpause the mashup whenever you want!
Replay the music!
training pipeline
How was the model made?
Training pipeline
Create dataset
Generate spectrograms
Extract best time stamps
Train model
Training pipeline
Create dataset
Generate spectrograms
Extract best time stamps
Train model
Create Dataset
Training pipeline
Create dataset
Generate spectrograms
Extract best time stamps
Train model
Generate spectrograms
Training pipeline
Create dataset
Generate spectrograms
Extract best time stamps
Train model
Extract best time stamps
Extract best time stamps
Extract best time stamps
Extract best time stamps
Training pipeline
Create dataset
Generate spectrograms
Extract best time stamps
Train model
Hyperparameter optimization
Loss
Epochs
Validation Loss
Epochs
Model architecture & Hyperparameters
256
2813
1407
1
128
4
176
16
16
256
256
240
Convolutional Layers
Dense Layers
Music video generation
Automated Music Video generation
Diffuse
Interpolate
Synchronize
Play
Gradually move from vector of one image to another for smooth transition
Input prompt to HuggingFace’s Stable Diffusion to generate images
Match the image frames to beats of music using spectrogram analysis
Utilize PyGame to play audio with images and save video
FUTURE IMPROVEMENts
Music Video GUI
Own playlist
Improve accuracy
Integrate music video generator into GUI & create prompts based on song lyrics
User identifies genre and preferences -> program creates new playlist
More memory and higher resolution spectrograms
Recap
Problem: Playing music without a DJ
Solution: Automated DJ With Cog*Ify
Generated Spectrogram
Extracted timestamps
Trained CNN
Deployed gui
Created music video