Pain points

	A	B	C	D	E
1	This was made in April 2022 related to the pain points survey for AI safety. Shared 2/6/2024.
2	Point	Title	Description	Solution / enhancement proposals	Other notes
3	Pain	Entry into AI safety is hard	The field of AI safety is on the surface relatively closed-off and restricted to a few specific institutions and groups.	- Ideas platform to easily get ideas for projects - Easy access to internships and volunteer projects
4	Pain	Culture of inaction for validation	There is a centralization of decision-making within EA where the community defers decisions to the thought leaders where most projects should just be started instead of delayed for validation.	- More support for action and hiring in local groups - Support for group development in Africa, India and SE-Asia
5	Pain	Missing a culture of celebration (culture of criticism)	There is often a culture of criticizing something before being excited for its execution or development. This part is similar to "Culture of inaction". Additionally, when projects go well, there is rarely any unsolicited positive reaction from the community.	- Having explicit celebrations of community victories at EAG(x)s - Providing rewards for some of the biggest problems - Having a dedicated celebrations group within EA
6	Plus	Allows for different thinking	The general thought process in EA and AI safety works with a relatively different thought process from the ones you normally see and this allows for new perspectives and interpretations of solutions.
7	Plus	Creates data-driven impact on the world	It is rare to see major communities so focused on making decisions based on data and this creates a whole new opportunity for maximizing impact.
8	Pain	Missing meeting place for AI safety research	The current best places to meet around AI safety are mostly online, private communities and organizations or EAG(x)s.	- Create a conference for AI safety research (already exists: 1, 2, 3)
9	Pain	Missing consensus about solution space in AI safety research	It is hard to navigate AI safety as an early career ML researcher because of the differing opinions on how impactful different strategies might be in AI safety research.
10	Pain	Loads of funding but it's really hard to get it	A lot of messaging about how much funding there is but inefficient, centralized funding causes slow processing and high standards based on ethos (see "Culture of inaction"). See also this post for a nice summary.
11	Pain	Instability of academic career	The academic career is generally unstable and does not allow for planning out your work-life balance nor long-term life decisions.
12	Pain	Lack of available supervision	Technical AI safety research (MIRI, ARC) requires mentorship to be aligned with the research.
13	Pain	Steps between learning AI safety and doing AI safety	There is a gap between taking a basics in AGI safety course and working within AI safety, e.g. should I do projects, work at Google or do something completely different? What is the next step in the onboarding into AI safety?
14	Pain	Missing scalable outreach	It is relatively easy to persuade people to join AI safety in 1-on-1s but this is not scalable. We need more ways to reliably get people into AI safety research.	- Youtube channels to point to, e.g. Rob Miles
15	Pain	Missing benchmarks and datasets	CV had ImageNet and MNIST, NLP has a hundred benchmarks, but AI safety only has very few. Creating benchmarks like TruthfulQA can be incredibly valuable.	- Create an org focusing on dataset creation
16	Pain	Missing clear rewards for solving AI safety problems	There's many relatively clear problems in AI safety that are not emphasized in the community nor the incentive structures of AI safety research.	- Millenium prizes
17	Pain	AI safety is generally pessimistic to work in	A bit like "Missing a culture for celebration", most people in AI safety have a pessimistic attitude to how much potential we have to deal with AI safety which can be seen as a net negative in the attempt to solve this problem since it excludes people.	- Learn the criticism sandwich
18	Pain	No good arguments against alignment being a problem and nobody incentivized to have them	Most arguments against alignment being a problem have generally been dealt with by Yudkowsky and/or are just not sophisticated enough. Nobody interested in the question are actually incentivized to Red Team the AI safety community's alignment focus. The best example we have is Paul Christiano.	- Have some core personnel focus only on Red Teaming for at least a few months
19	Pain	Top EA is too focused on AI safety	People outside AI safety in EA feel left out that AI safety is such a massive focus while only accepting a small subset of skilled talent capital.
20	Pain	LessWrong is significantly more negative than the EA Forum	This is another issue of culture. There's a lot more judgement on LessWrong and a vibe of "you're not saying anything new" compared to the excitement and encouragement of EAF.
21	Pain	AI safety is too far removed from AI capabilities research	Having a centralized community for AI safety research through the Alignment Forum and LessWrong is great but is subject to segregation from active research in capabilities that might 1) assist in improving AGI safety and 2) miss potential influence on the capabilities field.
22	Pain	AI safety is generally pessimistic to work in	A bit like "Missing a culture for celebration", most people in AI safety have a pessimistic attitude to how much potential we have to deal with AI safety which can be seen as a net negative in the attempt to solve this problem since it excludes people.
23	Pain	Definitions are unclear and the field lacks clarity as a result	Core researchers disagree on what the best ways to solve the alignment problems are and the difference in definitions do not help this problem.
24	Pain	The words for slow and fast takeoff are misleading	Slow takeoff will lead to the fastest onset AGI while fast takeoff will probably be later.
25	Pain	Formal definitions that are wrong are quite harmful	These mislead both future research and constrain our understanding of where we might need to target our efforts.
26	Pain	AI safety research often jumps over crucial reasoning steps	There is a tendency to imagine a series of steps that lead to a failure case and then go deep into that failure case while ignoring possible assumption limitations in the previous steps. Also related to "Researchers seem too focused on single failure modes".
27	Pain	Missing consensus about solution space in AI safety research	It is hard to navigate AI safety as an early career ML researcher because of the differing opinions on how impactful different strategies might be in AI safety research.
28	Pain	Lack of consensus among AGI researchers	The field of AI safety works a lot on the problems of alignment and have short timelines while AGI capabilities researchers generally have much longer timelines.
29	Pain	Field is dominated by MIRI theory	As the original forefront of AI safety research, MIRI's theoretical frameworks seem to dominate many AI safety researchers' perspectives on the field. This might be harmful for new ideas entering the field.
30	Pain	Researchers seem too focused on single failure modes	There is a problem of not knowing how probable different failure modes are and current researchers seem to be very focused on quite specific failure modes. This plays together with "No good arguments against AI safety".
31	Pain	No clear visualizations of how a slow takeoff will look from an X-risk perspective	We are currently missing clear perspectives on how a slow takeoff will look and put humanity at risk. CAIS is one attempt towards this.
32	Pain	No clear connections between ELK and the rest of the field	We should work on showcasing how ELK can assist or inform our work on other concepts in AI safety.
33	Pain	Missing a view of how far the field currently is	There is a general issue of keeping up with how far we are towards solving the alignment problem. Newer projects have been better at showcasing their value towards the solution but it is still an issue.
34	Pain	Unclear what the future path looks like	Is it an insights problem? Can we see incremental improvement? It would be nice with more clarity on these and similar questions.
35	Pain	Keeping up to date is hard	This is a general problem in research but would be ideal to work on in AI safety. Rob Miles is a good example for AI safety, Yannic Kilcher for AI capabilities, and Károly Zsolnai-Fehér for physics-based deep learning (PBDL).
36	Pain	We don't have many good decompositions of problems	ELK is a good example but most problems in AI safety requires people to understand the framings in a holistic way that necessitates a lot of interdisciplinary research understanding. If we can come up with better decompositions of problems, this problem might be alleviated.
37	Pain	Missing clear rewards for solving AI safety problems	There's many relatively clear problems in AI safety that are not emphasized in the community nor the incentive structures of AI safety research.
38	Pain	Missing a Theory of Change in AI safety	The big organizations mostly do not have specific plans for how we can properly do work on AI safety, why it is important and in which ways we can think about it.
39	Pain	Most EA forecasts are very fuzzy	Hard to weigh predictions and the predictions are quite disparate. Researchers don't agree and there's also not specific prediction markets about the decomposition of probabilities.
40	Pain	Not many independent hopes for how to do AGI well	The field of AI safety has very few perspectives on how AGI can end up working out well for the world. Examples might be truthful LLMs, ELK and CAIS while most scopes seem to be quite narrow.
41	Pain	Lack of available supervision	Technical AI safety research (MIRI, ARC) requires mentorship to be aligned with the research.
42	Pain	Missing feedback from the top researchers	There is a large need for good research taste and we might be able to get even more feedback from top researchers.	- Regular AMAs from top researchers to widely support research
43	Pain	It's very hard to not help capabilities research	Many of the contemporary and useful projects we do in AI safety research predicate on the strength of future models and need to simulate some sort of higher capability. This automatically incentivizes AI safety researchers to work implicitly on AI capabilities research.
44	Pain	Aligned models need to be just as capable as unaligned models	For future systems to utilize aligned models, our conceptual work needs to end up with models that are inherently better. This relates to the "It's very hard to not help capabilities research" pain point.
45	Pain	Relating to AI capabilities researchers	Too little done in reigning in current AI work or incentivize AI safety in AI capabilities. This both calls for more openness and more AI governance work.
46	Pain	It's hard to evaluate how good our proposed solutions are	We present a lot of different models but there is not a clear relationship between them nor with our vision of how it might turn out well. E.g. Mark Xu mentioned at EAG that solving ELK might get us 20-25% towards solving the alignment problem more generally. These sorts of quantifications are few and far between.
47	Pain	We are missing the tools to be able to evaluate current models	As it states, model evaluations are generally ad hoc or based off of datasets. We are missing these datasets for AI safety and/or even better tools for evaluations of alignment.
48	Pain	Missing benchmarks and datasets	CV had ImageNet and MNIST, NLP has a hundred benchmarks, but AI safety only has very few. Creating benchmarks like TruthfulQA can be incredibly valuable.
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100