Breaking into ML Research

Maxwell Jones

Flux dev, Black Forest Labs

Talk Overview

• Brief introduction of myself
• “Breaking Into” machine learning research
• What does success mean for an initial project
• What kind of skills are useful to gain before starting research
• Building good habits during an initial project(s)

My Background

My Background

• Currently a first year PhD student in the Machine Learning Department (MLD)
• Coadvised by Jun-Yan Zhu and Ruslan Salakhutdinov

Lab meeting with Jun-Yans lab

My Background

• Currently a first year PhD student in the Machine Learning Department (MLD)
• Coadvised by Jun-Yan Zhu and Ruslan Salakhutdinov
• Previously a Machine Learning Masters student at CMU

Lab meeting with Jun-Yans lab

Presenting Theory work at UAI

My Background

• Currently a first year PhD student in the Machine Learning Department (MLD)
• Coadvised by Jun-Yan Zhu and Ruslan Salakhutdinov
• Previously a Machine Learning Masters student at CMU
• Previous Previously an AI +Math Undergrad at CMU

Lab meeting with Jun-Yans lab

Presenting Theory work at UAI

Selfie with Martial Herbert (SCS Dean)

My Past Work

What to look for in an initial project

What to look for in an initial project

• Starting out, it’s good to generally have some research interests but don’t be too picky

Image: istockphoto.com

What to look for in an initial project

• Starting out, it’s good to generally have some research interests but don’t be too picky
• Many people are working on lots of things, and you might not even know what you’ll end up liking
• The people you initially want to work with may not even be looking for students

Image: istockphoto.com

What to look for in an initial project

• Starting out, it’s good to generally have some research interests but don’t be too picky
• Many people are working on lots of things, and you might not even know what you’ll end up liking
• The people you initially want to work with may not even be looking for students
• More important than any exact research fit is finding an advisor/PhD student that is invested in your success

https://www.apa.org/monitor/2019/04/mentor-ethically

Connecting with Possible Advisors

Dall-E, OpenAI

Connecting with Possible Advisors

• Talking to professors whose classes you are taking (ex. My first research project)

Connecting with Possible Advisors

• Talking to professors whose classes you are taking
• Doing well in the class can help you stand out
• You can get face time with them during office hours/after class
• Even if you don’t end up working with them, they can help refer you to other potential matches (and they can speak to how diligent you are as a student)

Connecting with Possible Advisors

• Talking to professors whose classes you are taking
• Asking your academic advisor to connect you with people (ex. My second research project)
• Faculty may be more likely to respond to an email from faculty than an email from a student

Kathleen Fu, in The Chronicle

Connecting with Possible Advisors

• Talking to professors whose classes you are taking
• Asking your academic advisor to connect you with people
• Asking friends that do research to connect you (ex. Many testimonies from undergrad friends)

Istockphoto.com

Connecting with Possible Advisors

• Talking to professors whose classes you are taking
• Asking your academic advisor to connect you with people
• Asking friends that do research to connect you
• Looking for Research Summer Programs (ex. Some members of our lab and friends)
• CMU RISS, Berkely Data Science Discovery, etc

Connecting with Possible Advisors

• Talking to professors whose classes you are taking
• Asking your academic advisor to connect you with people
• Asking friends that do research to connect you
• Looking for Research Summer Programs
• Emailing people (very tedious and slightly soul crushing, but worth it)
• Introduce yourself, short intro
• Why you’re interested in the lab
• Quick overview of relevant work (past GitHub projects or internships)
• Provide resume + (transcript if comfortable)
• Don’t ChatGPT or copy/paste an email (we know ☺)

Working Hard is Generally Valuable

• The better you do in key classes, the more a prof/advisor can give you a good reference
• If you can do good projects, your friends can talk about how great you did
• In general, putting out good work (even if it’s not exactly in the field you want) will pay dividends back

Vecteezy.com

Preparing for Research and Defining Success

https://www.deeplifejourney.com/

How to (prepare to) build good research habits

• Make sure you have a strong fundamental understanding of the basics
• Pay attention in Intro to ML (and other classes), since you’ll continuously need these skills
• Make sure to put in the time for the classes that will be beneficial to your career

How to (prepare to) build good research habits

• Make sure you have a strong fundamental understanding of the basics
• “You can just do things” – some guy on twitter once

How to (prepare to) build good research habits

• Make sure you have a strong fundamental understanding of the basics
• “You can just do things” – some guy on twitter once
• Do some end-to-end ML projects
• Doing simple training on small models or finetuning big models on a small amount of data is an extremely valuable skill

How to (prepare to) build good research habits

• Make sure you have a strong fundamental understanding of the basics
• “You can just do things” – some guy on twitter once
• Do some end-to-end ML projects
• Reproducing some cool results from a paper (harder than you might think)

How to (prepare to) build good research habits

• Make sure you have a strong fundamental understanding of the basics
• “You can just do things”
• Do some end-to-end ML projects – even doing simple training/finetuning from start to end is an extremely valuable skill
• Train a model or reproduce some cool results from a paper (harder than you might think)
• Do some ML competitions (kaggle or otherwise) – even if you don’t win, the experience and learning from other people’s submissions and reports is super valuable

How to (prepare to) build good research habits

• Make sure you have a strong fundamental understanding of the basics
• Pay attention in Intro to ML
• Make sure to put in the time for the classes that will be beneficial to your career
• “You can just do things” – do projects you find cool
• Push to Github, and try and make your code easy to use for others with comments and a README

How to (prepare to) build good research habits

• Make sure you have a strong fundamental understanding of the basics
• Pay attention in Intro to ML
• Make sure to put in the time for the classes that will be beneficial to your career
• “You can just do things” – do projects you find cool
• Push to Github, and try and make your code easy to use for others with comments and a README
• Some people may check your github before considering you as a candidate ☺

What is “Success” for an initial project?

• Try not to worry as much about the end result of a collaboration

James Clear

What is “Success” for an initial project?

• Try not to worry as much about the end result of a collaboration
• This is highly dependent on many factors out of your control (timeframe, randomness, etc)

James Clear

What is “Success” for an initial project?

James Clear

• Try not to worry as much about the end result of a collaboration
• Learning from who you’re working with is a better goal
• Consistently meeting with and learning from mentors (even if you have no progress)
• Building good research habits and being motivated to try your best

​

Good Initial Research Habits

India.com

Building good research habits initially

• Make sure you understanding key previous results (and can run baselines/other codebases)

Building good research habits initially

• Make sure you understanding key previous results (and can run baselines/other codebases)
• Make sure you write code that is easy to read and work from
• No “magic numbers”, name variables, name everything important
• Don’t make your code so modular its unclear for an outsider to know what’s going on (TODO: add Andrej karpathy blogpost)

Building good research habits initially

• Make sure you understanding key previous results (and can run baselines/other codebases)
• Make sure you write code that is easy to read and work from
• No “magic numbers”, name variables, name everything important
• Don’t make your code so modular its unclear for an outsider to know what’s going on
• Decrease your burden by writing functions that can be used over and over again
• Make it easy to try out many different experiments, without lots of overhead
• Try and work smarter (and also hard lol)

Building good research habits initially

• Make sure you understanding key previous results (and can run baselines/other codebases)
• Make sure you write code that is easy to read and work from
• Ask Questions!!!

Istockphoto.com

Building good research habits initially

• Make sure you understanding key previous results (and can run baselines/other codebases)
• Make sure you write code that is easy to read and work from
• Ask Questions!!!
• As a younger student, you have all the leeway in the world to pick the brain of the people you are working with
• Especially at the beginning, there are no stupid questions
• Every field has some implementation details that are hard to learn without someone mentioning them to you, so you have to ask
• Try to work in the same space as mentor figures if possible

Building good research habits initially

• Make sure you understanding key previous results (and can run baselines/other codebases)
• Make sure you write code that is easy to read and work from
• Ask Questions!!!
• Perseverance
• Research is hard and doesn’t work a lot, and can take a long time to get working (if ever)
• Implementation details can be as or more important that the method
• Perseverance (which can include pivoting) in the face of adversity is key to long term success

Key Takeaways

• Try and do well with current school/projects
• Reach out to people in many ways
• Prioritize good mentorship over exact research fit
• Maximize your learning from mentors in research

Questions??