A self-serving approach to living a good life as a researcher

Rosanne Liu

Google DeepMind, ML Collective

https://rosanneliu.com/

​

Deep Learning Indaba, 9 September 2023

A random bookstore at a random airport

A random bookstore at a random airport

An imaginary “Best Seller for ML Researchers”

How to code

How to form ideas

How to write

How to build a network

How to speak

How to read papers

What’s demanded in Today’s ML World

How to code like a 10x engineer

How to form groundbreaking ideas

How to write really good papers

How to build a network with famous people

How to speak like a cult leader

How to read papers really fast

What’s demanded in Today’s ML World

How to code like a 10x engineer

How to form groundbreaking ideas

How to write really good papers

How to build a network with famous people

How to speak like a cult leader

How to read papers really fast

Read

Code

Idea

Write

Speak

Network

…

What’s demanded in Today’s ML World

Skills

💯

Read

Code

Idea

Write

Speak

Network

Let’s say this gets a researcher to a somewhat content place

…

But we all start from different baselines

But we all start from different baselines

Maybe I’m a normal PhD student from a non-fancy school

But we all start from different baselines

Skills

💯

Read

Code

Idea

Write

Speak

Network

Maybe I’m a normal PhD student from a non-fancy school

…

But we all start from different baselines

Skills

💯

Read

Code

Idea

Write

Speak

Network

Maybe I’m a normal PhD student from a non-fancy school

…

But we all start from different baselines

Skills

💯

Read

Code

Idea

Write

Speak

Network

Maybe I’m a self-taught engineer from a non-English speaking country

But we all start from different baselines

Skills

💯

Read

Code

Idea

Write

Speak

Network

Maybe I’m inherently not a confident person

But we all start from different baselines

Skills

💯

Read

Code

Idea

Write

Speak

Network

Maybe I’m just an introvert…

We seem to have a single gold standard for what leads to success

💯

💯

💯

💯

💯

💯

But some are born closer to the finish line

💯

💯

💯

💯

💯

💯

Which means “equal chance for all” simply doesn’t exist

💯

💯

💯

💯

💯

💯

Apart from the “legitimate” standards, we also harbor absurd biases.

Be in a historically favored race & gender

Walk around with a lot of confidence

Have a low voice

Have a simple, catchy name, like John Wick

Be tall

(if you are a man)

Be pretty but not too pretty

(if you are a woman)

…

Well…

“Life isn't fair, it's just fairer than death, that's all.”

― William Goldman, The Princess Bride

​

But I am here to argue that it could be better

But I am here to argue that it could be better

Because the gold standard is not a ground truth; it is a happen-to-be.

​

But I am here to argue that it could be better

Because the gold standard is not a ground truth; it is a happen-to-be.

This is how it happened:

​

1. There was nothing

2. Some people got ahead by pretty much pure luck

3a. They opened doors for people who are like them

3b. Their arbitrary, uncorrelated traits get sold as as “the way to be”

4. The traits get reinforced and become the golden standard

Structure of today’s talk

• A brief recipe for success in ML research (current status quo)
• A self-serving approach to success
• A bit about ChatGPT
• Enough about big picture, what ML research topics should one work on?

Structure of today’s talk

• A brief recipe for success in ML research (current status quo)
• A self-serving approach to success
• A bit about ChatGPT
• Enough about big picture, what ML research topics should one work on?

Structure of today’s talk

• A brief recipe for success in ML research (current status quo)
• A self-serving approach to success
• A bit about ChatGPT
• Enough about big picture, what ML research topics should one work on?

How to code like a 10x engineer

How to form groundbreaking ideas

How to write really good papers

How to build a network with famous people

How to speak like a cult leader

How to read papers really fast

Read

Code

Idea

Write

Speak

Network

a collection

a recipe

Instead of

we have

1. Born in the West

2. Speak English natively

3. Have family ties with Bill Gates

4. …

Recipe for a stellar scientist candidate

• Education record: elite school, famous advisor, flawless grades
• Internship experiences before graduating: top industrial labs; 3-4 of them
• Publication record:
• First, lots of solid, first-author papers (plus if with famous last authors)
• Then, a number of middle-author papers to show that you are “collaborative”
• Finally, moving on to a few last-author papers to show your “leadership”
• A series of papers working on the same topic, establishing you as “the expert,” before branching out to show that you also have “broad interests”
• Plus if a recognizable name on twitter

Recipe for a stellar scientist candidate

• Education record: elite school, famous advisor, flawless grades
• Internship experiences before graduating: top industrial labs; 3-4 of them
• Publication record:
• First, lots of solid, first-author papers
• Then, a number of middle-author papers to show that you are “collaborative”
• Finally, moving on to a few last-author papers
• A series of papers working on the same topic, establishing you as “the expert”, before branching out to show that you also have “broad interests”
• Plus if a recognizable name on twitter

😈: What’s wrong with this? We’ve been hiring like this for decades and they’ve all turned out great.

Yes, but

• Survival bias: others just as great but don't fit the rubric never got a chance to be proven right or wrong
• Misconception: So much of what’s considered “talent” is “privilege” and the accompanied “opportunity access”
• When we converge, we don’t train/learn

Structure of today’s talk

• A brief recipe for success in ML research (current status quo)
• A self-serving approach to happiness
• A bit about ChatGPT
• Enough about big picture, what ML research topics should one work on?

The self-serving approach

Vision:

What if we can just freely be who we are, with all of our strengths and weaknesses, grow however we want, and have equal or quasi-equal chances to opportunities?

Zootopia, 2016 film, https://twitter.com/DisneyAnimation/status/1154064076609167361

The self-serving approach

Approach:

1. (top-down) change the standard!

The self-serving approach

Approach:

• (top-down) change the standard!

How do we challenge a system that’s made for one kind?

• I never trust that humans can be bias-free; I think favouritism is strongly initialized in us and hard to mitigate. The best we can do is improve representation and diversity in positions of power
• If you are in any way non-standard and made it somewhere, remember to open doors for others like you
• Providing the first opportunity to someone, making a 0-to-1 change in someone’s life will be the most rewarding thing you ever do

The self-serving approach

Approach:

2. (bottom-up) collaboration!

The self-serving approach

Approach:

2. (bottom-up) collaboration!

How to find collaborators (as a brand new person)

• (formal) Employment: internship
• (casual) Networking at conferences, meetups, online platforms, and

​

​

How to find collaborators (as a brand new person)

• (formal) Employment: internship
• (casual) Networking at conferences, meetups, online platforms, and

​

​

A dating site mingling space for ML researchers looking for assistance (collaboration, mentorship, compute grant, etc.) , to hang out and to grow at their own pace.

https://mlcollective.org

​

How to find collaborators (as a brand new person)

• (formal) Employment: internship
• (casual) Networking at conferences, meetups, online platforms, and

​

​

A dating site mingling space for ML researchers looking for assistance (collaboration, mentorship, compute grant, etc.) , to hang out and to grow at their own pace.

Purely voluntary.

https://mlcollective.org

​

How to find collaborators (as a brand new person)

• (formal) Employment: internship
• (casual) Networking at conferences, meetups, online platforms, and

​

​

A dating site mingling space for ML researchers looking for assistance (collaboration, mentorship, compute grant, etc.) , to hang out and to grow at their own pace.

Purely voluntary.

Free to join.

https://mlcollective.org

​

I spend a lot of time thinking about outside-of-employment collaborations

• The kinds
• The mentor-mentee kind
• The all-peers kind
• How to get started
• When to commit
• How to maintain the commitment
• How to strengthen the tie when it’s purely voluntary

I spend a lot of time thinking about outside-of-employment collaborations

• The kinds
• The mentor-mentee kind
• The all-peers kind
• How to get started
• When to commit
• How to maintain the commitment
• How to strengthen the tie when it’s purely voluntary

Mentor pitches idea, mentee implements

Mentee pitches idea and prelim result, mentor advises

I spend a lot of time thinking about outside-of-employment collaborations

• The kinds
• The mentor-mentee kind
• The all-peers kind
• How to get started
• When to commit
• How to maintain the commitment
• How to strengthen the tie when it’s purely voluntary

Mentor pitches idea, mentee implements

Mentee pitches idea and prelim result, mentor advises

RFPs: https://mlcollective.org/rfp

​

Research Jam: https://mlcollective.org/events/#jam

​

​

I wanted “work” to be like making friends

“The same thing that makes friendship so valuable is what makes it so tenuous: it is purely voluntary. You enter into it freely, without the imperatives of biology or the agenda of desire. Officially, you owe each other nothing.”

― Tim Kreider, We Learn Nothing

ML Collective (MLC) is my self-serving approach to be happy

• I’m not your standard candidate, or have a killer resume at any point in my career
• EECS undergrad, but never even had a computer before entering college
• 2 PhD attempts, 1 ~success
• A few papers, but they are more “cute” than groundbreaking
• We filmed funny “paper explanation” videos
• We are very very serious about the writing of papers
• But why am I your keynote speaker here?
• I did a few things not following any recipes, necessarily, but feel authentic to me
• Followed my strength rather than weighed down by my shortcomings
• “The authentic career”

My wildest dream

At this point, the most authentic-feeling career for me

​

​

​

How to stay authentic

• Follow your advantage
• If you are already weird, be weirder
• Resist the temptation to assimilate
• Share your story; influence others to live truly

​

​

Aside: what if being myself is bad? 😈

• What if I'm intrinsically a bad person? What if what makes me happy is to see others fail?
• I believe the need to serve and foster a community is intrinsic.
• Any objectives away from that is instilled by an unhealthy environment.

​

​

Structure of today’s talk

• A brief recipe for success in ML research (current status quo)
• A self-serving approach to success
• A bit about ChatGPT
• Enough about big picture, what ML research topics should one work on?

What the future of research is like (with ChatGPT and everything): the optimistic version

Skills

💯

What I have

What the future of research is like (with ChatGPT and everything): the optimistic version

Skills

💯

What I have

What the future of research is like (with ChatGPT and everything): the optimistic version

Skills

💯

What I have

AI assistance

What the future of research is like (with ChatGPT and everything): the optimistic version

• You’d be asked to take ownership much sooner
• You’d perform the role of lab manager / PI from very early on
• You’d have to act entrepreneurially much sooner

Structure of today’s talk

• A brief recipe for success in ML research (current status quo)
• A self-serving approach to success
• A bit about ChatGPT
• Enough about big picture, what ML research topics should one work on?

Remember “use your advantage”? Your biggest advantage right now is that you have nothing to lose.

• You can work on ML topics that others couldn’t afford to work on.
• Pre-training improvements
• Alternative training objectives
• Alternative training recipes

​

​

The recipe everyone is talking about now

Andrej Karpathy, How to train your Chat(GPT) Assistant: an emerging recipe, 2023

The recipe everyone is talking about now

Andrej Karpathy, How to train your Chat(GPT) Assistant: an emerging recipe, 2023

But it’s not a golden standard! It’s a happen-to-be.

Remember “use your advantage”? Your biggest advantage right now is that you have nothing to lose.

• You can work on ML topics that others couldn’t afford to work on.
• Pre-training improvements
• Alternative training objectives
• Alternative training recipes
• ML world right now is full of hacks and tricks; and no one has time to understand why. But you can bring more science to it.
• Follow your aesthetics

​

Structure of today’s talk

• A brief recipe for success in ML research (current status quo)
• A self-serving approach to success
• A bit about ChatGPT
• Enough about big picture, what ML research topics should one work on?

A self-serving approach to living a good life as a researcher

Thank you!

Rosanne Liu

Google DeepMind, ML Collective

https://rosanneliu.com/

​