DeepOculus: Generative AI For Eye Retina Disease Tracking

By: Timothy Gao, Henry Hong, Marcus Koh, and Ohm Rajpal

Our Team

Timothy Gao

Ohm Rajpal

Marcus Koh

Henry Hong

ABOUT THE CHALLENGE

01

The Challenge

Use generative AI to create artificial retinal video sequences & eye motion traces given a disease state.

Impact

02

Understanding eye diseases

Using Generative AI, doctors can understand how the an average retina would look when affected by a particular disease.

Doctors can use this to help diagnose patients, learn what causes the disease, and/or how to effectively treat the disease.

Our Product

03

DeepOculos 1

Introducing

What is Deep Oculos 1?

LSTM Artificial Eye Movement Generator

GAN Retinal Video Generator

Disease state

MS

mTBI

MCI

Example

Show me what a patient’s retina would look like if they are affected by mTBI

Thank you DeepOculos!

Here you go!

​

​

​

​

​

Beep Boop Beep

How We Built It

04

• Created time series based on vertical and horizontal eye movement data

​

​

​

​

• Long Short-term Network captures long term and short term patterns in past eye movement data to predict and generate future eye movement patterns

LSTM Architecture

• Used a four-layer LSTM network with dropout regularization after each LSTM layer and a single dense neuron output layer
• Trained over 50 epochs, with rmsprop optimizer and mean_squared_error as loss function
• Utilized first 80% of each time series as training data, last 20% as testing data
• Used MinMaxScaler to normalize all pixel values to [-1, 1] domain
• Used previous 60 frames as context for each sliding window

​

Implementation Details

Model Results

DCGAN (Deep Convolutional GAN)

• Combined Convolutional and Transposed Convolutional layers into GAN architecture
• Sample from Gaussian to generate initial random state

​

​

​

​

​

​

​

​

​

​

​

• Extracted the video frames using a parser
• Downsize video frames from 512x512 to 64x64
• Created discriminator using positive values for real retinal data and negative values for fake data
• Generates video frames, which can then be turned into a video sequence

Convolutional Layer

• Extracted every other video frame
• Downsize video frames from 512x512 to 64x64

DCGAN

Original 512x512

​

​

​

​

Downsized 64x64

• Created discriminator using positive values for real retinal data and negative values for fake data
• Used LSTM-generated eye movement data to generate artificial retina video frames
• Combine all 64x64 video frames into video

​

DCGAN (Deep Convolutional GAN)

Generator Model Summary

Discrimater Model Summary

Combined GAN Architecture

Training for 1000 Epochs

2000 Epochs

3000 Epochs

Generated

Real

Generated

Real

Final Result

Technologies Used

05

Technologies Used

• Tensorflow: To build LSTMs, GANs
• Scikit-Learn: MinMaxScaler and Train Test Split
• Keras: Deep learning API (Sequential Model and neural network layers)
• Seaborn and Matplotlib: Data Visualization
• NumPy: For calculations and array manipulation
• Pandas: For processing CSV files

​

Challenges Faced

06

Challenges

• GPU was not powerful enough to run on 512x512 images as computation scales exponentially
• Inefficient code for ML with large datasets
• Difficult to traverse through huge data set
• Autoencoder failed due to large image size and the images were too large for processing (512 x 512)
• We downscaled and gray scaled to (64 x 64)

​

​

Future Developments

• Use GAN to generate full-scaled 512x512 images
• Further tune our model to better bridge the connection between our LSTM and GAN models
• Transitioning from average results to patient-oriented results
• Ophthalmoscopes are expensive

Thank you! Q/A

07