ML Collective

Town Hall

10/24/21

Welcome 🎉

Agenda

• Everyone say a quick hello!
• A few slides on some specific matters
• Open discussion with notes
• Hear general thoughts
• Brainstorm ideas together
• Assign any actions

What is MLC

A research organization

(we publish papers!)

A non-profit company

A community

A coworking space

A study group

A career network

MLC is a lab

for those who want

a research home.

Target audience: non-traditional researchers

But really, anyone who’d find us useful.

How it started

2018.6

A reading group

2020.6

A bunch of us continued to have lab meetings even though we no longer worked together

2020.12

Discord server started after a successful NeurIPS Social.

2020.8

2017.1

Deep Collective group at Uber

Renamed “ML Collective”

How it started

2018.6

A reading group

“MLC: Open Collab”

2020.6

A bunch of us continued to have lab meetings even though we no longer worked together

2020.12

Discord server started after a successful NeurIPS Social.

2020.8

2017.1

Deep Collective group at Uber

“MLC: Reading”

“MLC: Lab”

Renamed “ML Collective”

How it started

2018.6

A reading group

2020.6

A bunch of us continued to have lab meetings even though we no longer worked together

2020.12

Discord server started after a successful NeurIPS Social.

2020.8

2017.1

Deep Collective group at Uber

“MLC/events”

“MLC/community”

“MLC/projects”

“MLC/events”

“MLC/community”

“MLC/events”

“MLC/projects”

Renamed “ML Collective”

How it’s going

• Discord: 2090 members*
• DLCT Reading group: 2086 subscribers*
• Twitter: 4,950 followers*
• YouTube Channel: 739 subscribers*
• Tons of events: research jams, office hours, OpenClubHouse, conf socials …
• 6 sponsored (GCP credits) projects
• A wiki (collaborative writing space)
• Now this very first Town Hall!

* as of Oct 23, 2021

How it’s going (cont.)

• Interest Groups!
• Physics informed ML
• Computer vision
• Natural language processing
• nlp_study_group
• Explainable ai
• ML theory
• Ethical AI
• Reinforcement Learning
• RL implementation
• Graph learning
• OOD and Uncertainty Quantification

​

Let’s hear from the interest groups!

Section 1: Interest Groups

Interest Group: #physics-informed-ml

Insert your slides here :)

End!

Looking forward to seeing all of you!

First meeting 5th Nov

Interest Group: #computer-vision

Motivation

Another reading group

17 sessions later...

Dynamic Role Playing

Question: How do we transition a non-traditional researcher into conducting research?

​

Philosophy:

1. Understanding research
2. Reproducing research
3. Producing research
1. Encouraging collaboration between ideas

Moving Beyond Reading Sessions

1. Personal Projects Workshop

2. Scene Bias Workshop + CSS

3. ICLR Blog Post Track,

Reproducibility Challenge*

We have 6-7 teams!

*Ongoing

Join us!

Interest Group: #reinforcement-learning

MLC Reinforcement Learning Reading Group

RL Mods: David Murphy, Shreyans Jain, Perusha Moodley

RL Interest Group - Establishment and Evolution

• Established as Paper Reading Group
• Followed the Role-based Model of Alex Jacobson and Colin Raffel
• See Alex Jacobson and Colin Raffel blog post https://colinraffel.com/blog/role-playing-seminar.html
• Held 5 Reading Sessions to date
• Successful first steps
• Content prepared by role players was good, educational for all
• Positive, interactive group dynamics
• Sessions have reasonable attendance
• But we hope for more
• We fall short on role coverage
• Difficult to span group needs
• RL is broad - hard to cover all interests
• Experience level varies widely

RL Interest Group - Addressing the Challenges (role playing)

​

• Well defined roles guide process
• Develops broad set of skills required for understanding papers
• Good learning experience - opportunity to focus on individual aspects of interpreting a paper

• Doing a role justice takes a significant amount of time (5-15 hours)
• Many people not confident taking on a role
• The Hacker role in RL is more complicated than other disciplines
• Interfacing to or creating an environment
• Learning algorithm
• Role playing unsustainable without large group (same people taking all roles each time)

​

Pros

Cons

RL Interest Group - Addressing the Challenges (group evolution)

​

• Feedback from the group members
• Hands-on sessions to develop skills and a deeper understanding of RL
• Encourage members to present their own research
• Broaden topic areas to libraries or other non-paper topics
• Encourage those passionate about a paper to present it
• Decoupled hacker role from role based paper presentations
• Evolution: adopt a two track approach
• Separate out hacker group
• Move to traditional paper reading format driven organically by group and member interest

RL Interest Group - Implementation (aka Hacking) Group

​

• Beginnings and Aspirations
• Take small, but consistent steps for growth
• Implement algorithms from important papers
• Approach implementation in a small team (3-5 people) setting
• Small teams meet regularly to share knowledge and experience
• Graduate to SotA papers, Conference challenges, Reproducibility Challenges
• First implementation cycle in process:
• Selected a group target - Implementation of A3C
• Divided into two teams.
• Fastest moving team has meet 5 times so far with great progress
• One group session to read the paper
• Four pair programming sessions
• Three larger meetings so far to compare notes among teams

Interest Group: #graph-learning

Graph Machine Learning:

IG Presentation @

MLC Town Hall Meeting

The Beginning of #graph-learning Channel

1. After having participated in the OGB challenge at KDD, we created the #graph-learning channel to discuss the graph-related papers at MLC
2. We also wanted to create a reading group to foster discussion and future competitions and eventually support research collaborations in GraphML
3. The #graph-learning channel was created

The Beginning of GraphML Reading Group

• After the competition finished, we wanted to start the reading group
• We sent a poll to understand how many people are interested and their topics of interest in GraphML
• After the poll, most of the participants wanted to cover the basics
• Thus we decided to start with an intro GraphML course, CS224W: Machine Learning with Graphs(Stanford University)

​

Current Activities

• The Reading Group meets every Saturday
• Two lectures are discussed on a weekly basis
• A safe place for beginners to ask and discuss the lectures
• Research papers are presented during the discussion
• We have different roles(role-playing) and anybody can volunteer

The Future of GraphML Reading Group

• Create notes about the course to facilitate learning and review at your preferred time
• Discuss papers(Theory+Applictions) on a constant basis
• Hands-on experience to get familiarized with graph frameworks
1. Do the homework from CS224w
2. Execute and understand code from a specific research paper being discussed

Why should you care about GraphML

• Computational Biology(Protein Structure Prediction)

​

Why should you care about GraphML

• Computer Vision (Point Cloud Processing)

​

Why should you care about GraphML

• Natural Language Processing

​

​

​

Why should you care about GraphML

• Reinforcement Learning(Routing Optimization)

​

​

​

Interest Group: #natural-language-processing

MLC NLP Reading Group

NLP Mods: Aiswarya Sankar, Ashutosh Kumar

Current Activities

Weekly meetings Saturday 10am PST

​

• Reading group with 1 paper a week
• Working on an ICLR blog post
• Workshops for paper feedback
• Since April 2021

​

​

Weekly study group meetings Tuesday 8pm PST

• Took CMU NLP 114 Course together
• Working on individual research proposals
• Group background research documentation for idea sharing
• Has been going on for the last 2 months

Highlights

Topics Covered so far!

​

• Paper list
• May: Seq2Seq models
• June: Multi document summarization models
• July: Topic modeling
• August: Conditional text generation
• September: Knowledge base text generation
• October: Non- autoregressive text generation

​

​

2 papers submitted to ICLR!

​

• ACM: Attribute Conditioning for Multi Document Summarization - Aiswarya Sankar
• Optimizing Binary Neural Networks - Adithya Venkateswaran

Interest Group: #ml-theory

ML theory

Date of birth : October 6, 2021

​

Method of birth : Deep learning group repurposed by Jatin Batra

Source: parents.org

Questions we care about

Factors controlling generalization

Geometric deep learning

Mikhail Belkin, Daniel Hsu, Siyuan Ma, Soumik Mandal, 2018

​

Michael Bronstein, 2021

Our luck to have help from an expert

Behnam Neyshabur

Staff scientist

Google

​

Available for office hours! For details, check the MLC events webpage.

Source: Twitter

Upcoming events

Streaming of this year’s course - starting week of Nov 1

​

​

Check pinned post in #ml-theory on discord for details

​

​

​

​

What are we discussing on #ml-theory?

Jim Belk, 2008

Mystery of generalization

Vaishavh Nagarajan. 2019

Interest Group: #ood-and-uncertainty-quantification

(A Few) Topics That Interest Us

• Identifying out-of-distribution instances rather than being confidently incorrect
• No measurement is complete without error bars: deriving believable PDFs instead of point estimates
• Evaluating and fixing calibration
• Dealing with long-tails

Exemplar papers

• A Tale Of Two Long Tails, D'souza et al. https://arxiv.org/abs/2107.13098
• Improving Uncertainty Calibration of Deep Neural Networks via Truth Discovery and Geometric Optimization https://arxiv.org/abs/2106.14662
• Generalizing to Unseen Domains: A Survey on Domain Generalization https://arxiv.org/abs/2103.03097
• Cross-validation: what does it estimate and how well does it do it? https://arxiv.org/abs/2104.00673
• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization https://arxiv.org/abs/2006.16241
• Out-of-domain Generalization from a Single Source: A Uncertainty Quantification Approach https://arxiv.org/abs/2108.02888

​

​

​

Ongoing Event

• SHIFTS Weather Prediction Challenge, at Bayesian Deep Learning workshop, NeurIPS '21: https://research.yandex.com/shifts/weather
• Use weather data (features) spanning latitudes +x to -x degrees, and 0 to t1, t2 to t3 times of the year, to make temperature predictions for latitudes +x+y to -x-y degrees and t1 to t2 time of the year
• Submitted workshop paper -- Sankalp Gilda, Neel Bhandari, Wendy Mak, Andrea Panizza

​

Future Events/Goals

• Weekly/bi-weekly reading group with role-playing
• Identify and take online courses
• Collaborate on and submit more papers
• Suggestions and participation welcome!

Collect feelings and ideas and discuss:

MLC Town Hall notes 10/24/21

Section 2: Discord

Discord organization

Discord organization

• Any new channels needed?
• Research-in-progress (I made a plot)
• Draft-review or draft-proofread or last-min-writing
• Post-your-own-wins or celebrate-wins
• A new category for mentorship/education related advice
• Maybe a meme channel
• A channel for programming/technical related questions

​

Discord roles

• 22 Current roles!
• Unknown: do we need RG_?? Channels?
• Can people just @channel?
• And only interested people join that channel?

Discord roles

• Old roles
• Superadmin: Jason and Rosanne
• Admin:
• Manage channels and roles
• Mention @everyone, @here, all other roles
• Delete or pin messages
• Organizer
• Mention @everyone, @here, all other roles

Discord roles

• New roles
• Superadmin: Jason and Rosanne
• Admin: (<- with more people here)
• Manage channels and roles
• Mention @everyone, @here, all other roles
• Delete or pin messages
• Admin-Github:
• Can make GH repos (volunteers here?)
• Admin-Overleaf:
• Can make overleaf projects (volunteers here?)
• ​
• Actual tasks admins seem useful for
• Making new channels (but we have no channel policy)
• Changing channel names
• Separate
• Uploading YouTube videos (just Rosanne for now)

Actions and responsibilities

• Post event announcement reminders in Discord
• Enable notifications for the appropriate channels

Discord bots

• We have these bots:
• Dyno
• Carl
• Do people find them useful?
• Would others be useful?

​

Collect feelings and ideas and discuss:

MLC Town Hall notes 10/24/21

Section 3: Everything else

You

• Where are you these days? What do you need?

A few open questions

• Should we make any changes to the organizing structure?�
• More localized Interest groups are working well. Which if any “all-hands” style meetings should we have?�
• How can we encourage more drafts, submissions, publications?
• Should we have KPIs?�
• Can we make wins more visible?�
• Should we do anything to encourage and maintain attendance?

​

MLC needs more mentors

• Not having a robust pool of mentors is a huge limiting step for MLC
• Is it the biggest bottleneck for our throughput of research?
• Does this make it the biggest bottleneck for people’s careers?
• Allow large range of mentors or “research buddies”?
• Super senior mentor
• ...
• Peer-level research buddy (who has been around MLC a bit longer and knows about research jams, etc).

MLC Wiki

• We have a wiki: https://mlcollective.org/wiki/

​

MLC Github

• What it’s for
• Who can make new repos
• Need some admins here to add people to our Org
• People in org can make repos and teams themself! But need an admin to add them via message to #help-github channel.
• Process for making repos, etc.
• https://mlcollective.org/wiki/how-to-use-mlc-github/
• Volunteers to try out the process, make a repo, document?

Hosting Events

• Playbook: https://mlcollective.org/wiki/host-mlc-event/
• Before event:
• Make page for event (wiki or other)
• Add event to MLC calendar
• Publicize event
• Tweet and tag @mlcollective for RT
• <anything else you want>
• Host event: <how to the event itself>
• (optional) Record event if you want (and get permission from people)
• (optional) Take notes
• After event
• (optional) Post video recording
• (optional) Post notes on wiki or other page

Collect feelings and ideas and discuss:

MLC Town Hall notes 10/24/21

At end:

MLC Hangout in Gather Town