A Design Laboratory:
Language and Language Models
Karina Nguyen
Twitter Email
Jacobs Institute for Design Innovation at UC Berkeley
October 24, 2023
A Design Laboratory:
Language and Language Models
Jacobs Institute for Design Innovation
October 24, 2023
Language and cultures of writing.
Language and cultures of writing.
“Language is one of the most interesting laboratories to study things.” – Dario Amodei
Language and cultures of writing.
Language evolves along with culture. New words are created to describe new concepts, technologies, social changes etc.
Language can shape and influence the way people in a culture think and perceive the world, a concept known as linguistic relativity or the Sapir-Whorf hypothesis. For instance, the presence or absence of certain vocabulary can emphasize or downplay certain concepts or feelings, thereby molding perceptions and behaviors.
Language as a high end product.
Most effective way to get people to read more of storylines coverage is by adding a layer of context to our product and coverage — across major surfaces — that connects people to the stories and information they need.
Elections 2020
Language as a high end product.
The goal is to increase the number of elections readers who read 2+ stories by 5% overall. This would lead to about 100,000 more readers a day reading a second elections article.
Context Blocks
Language as a high end product.
Language as a high end product.
Journalist Workflow
Language as a high end product.
iOS Widgets for Breaking News
Language as a high end product.
Engineering Catalog
Language as a committed way of showing love to users
Language as a committed way of showing love to users
Dropbox
Language as a committed way of showing love to users
Square – Virtual Terminal
Language as a committed way of showing love to users
Square
Language as the most compressed artifact of human knowledge
100K Context
Language as the most compressed artifact of human knowledge
Language as the most compressed artifact of human knowledge
Claude.ai – File Uploads
Language as the most compressed artifact of human knowledge
Language as the most compressed artifact of human knowledge
Claude.ai – Auto-generating Titles
Language as the most compressed artifact of human knowledge
Language as the most compressed artifact of human knowledge
Model Safety Card – Claude 2
Case Studies
Audience: policymakers, journalists, developers, academics
Main story: Claude 2 was an incremental improvement / evolution from previous model that steered towards what people wanted / user needs
Details
Claude Instant 1.2
Case Studies
Audience: developers, companies, journalists
Main story: cheaper, faster, better and worse than Claude 2 (cost-effective)
Note: 80/20, no model card but lots of qual. evals
Measuring Subjective Global Opinions in LLMs
Case Studies
Audience: academics, policymakers
Main story:
Details
Interactive data vis: llmglobalvalues.anthropic.com
Methods
Time
Heatmap
Legend
Process
Measuring Subjective Global Opinions in LLMs
Measuring Subjective Global Opinions in LLMs
Discovering LM’s Behavior Using Model-Written Evals
How I use Claude for research storytelling
Audience: academics, developers, policymakers
Main story: Model-written evals are as diverse and high quality as human manual evals but cheaper and more scalable. We generated ~150 evals and find that LMs are “sycophantic” due to RLHF and as the model size increases.
Details
Evaluations
Main story: Diversity within each dataset
Details
How I use Claude for research storytelling
How I use Claude for research storytelling
Language as the most compressed artifact of human knowledge
“If Stripe is a monstrously successful business but what we make isn't beautiful and Stripe doesn't embody a culture of incredibly exacting craftsmanship, I'll be much less happy.
My intuition is that more of Stripe's success than one would think is downstream of the fact that people like beautiful things. Because what does a beautiful thing tell you?
Well, it tells you the person who made it really cared."”
Patrick Collison
Thanks! Questions?
Jacobs Institute for Design Innovation UC Berkeley
October 24, 2023