2024 in Open Models
Luca Soldaini, Nathan Lambert
2023
2024
Llama 1&2
Mistral
MPT
Falcon
Yi
Llama 3
Qwen 2.5
Deepseek
Cohere ⌘R
DBRX
OLMo
Mistral
LLM 360
DCLM
Gemma
Why Open Models?
the "Just use model APIs" argument
Research�Thrives �Open Models
Modeling
Behavior
Inference
Evaluation
Open Models�Outperform �use cases
Retrieval (MTEB)
Edge AI
Community Stretches Beyond Models
https://qwenlm.github.io/blog/qwen2
Collaboration�When we're free to participate, we can enhance each other's work in unanticipated ways.
https://opensource.com/open-source-way
2024: Open Source AI definition
https://opensource.org/ai
https://www.interconnects.ai/p/defining-open-source-ai
Good��Most of the definition is very standard. Weights need to be available, code for using the system must be released under an OSI-approved license, etc.
Compromise��Vague on data��... sufficiently detailed information about the data used to train the system, so that a skilled person can recreate a substantially equivalent system…
We know way more than in 2023
We are more resource constrained than in 2023
2023
2024
Llama 1&2
Mistral
MPT
Falcon
Yi
Llama 3
Qwen 2.5
Deepseek
Cohere ⌘R
DBRX
OLMo
Mistral
LLM 360
DCLM
Gemma
Compute Rich Club�membership hotly debated
Pre-train
1k-∞
H100
Post-train
8-10k
H100
Inference
1-100
H100
Evaluation
0-100
H100
Research Budget
2023
2024
Llama 1&2
Mistral
MPT
Falcon
Yi
Llama 3
Qwen 2.5
Deepseek
Cohere ⌘R
DBRX
OLMo
Mistral
LLM 360
DCLM
Gemma
Fully Open Models�Hi! That's me.
Not just a checkpoint�the full recipe
https://arxiv.org/pdf/2409.02060
Any part of pipeline can be your starting point
Any part of pipeline can be your starting point
https://allenai.org/blog/olmo
Shameless plug�Ai2 fully open recipes
Open models in peril
It's not just compute
Harder to get data via crawling
Longpre et. al. 2024. Consent in Crisis: The Rapid Decline of the AI Data Commons. Data Provenance Initiative.
Widening inequality in data access
lobbying�open models = danger
lobbying�open models = danger
ignores efforts making open models safe
lobbying�open models = danger
ignores efforts making open models safe
brushes unfounded concerns under rug
PERSONAL OPINION�NOT MY EMPLOYER'S
I think we dodged a bullet 😊
Luca Soldaini��Twitter / X: @soldni
Bluesky: @soldaini.net�Email: lucas@allenai.org
Nathan Lambert��Twitter / X: @natolambert
Bluesky: @natolambert.bsky.social�Email: nathan@natolambert.com