Catherine Brewer, last updated May 16, 2025. Link sharing is on.

Externally shared. Internal version here.

Red lines for cyber evals

  • What this is:
  • We have an abundance of cyber evals and a lack of consensus on what results (if any) would be sufficiently scary to constitute a “red line”. (Also, we don’t agree on what such a red line would look like.)
  • As with bio red lines, I imagine this would involve gathering a bunch of important stakeholders from industry and maybe govts, having a bunch of conversations with them, and getting some consensus around (a) unambiguously big-deal eval scores and (b) the rough kind of response that would be merited. You can then publish a splashy consensus paper, or a
  • Why:
  • Same rationale as with bio red lines: one big way evals fail is the frog-boiling/no consensus on ifs nor thens.  
  • Who could do it:
  • Probably you would have to be fairly senior/professional-seeming (senior enough to get people to reply to your emails etc, professional-seeming for the project to work) and good at interfacing with experts.
  • I don’t think you would need to be a domain expert, but having thought about cyber evals a fair bit and being proficient in the cutting edge there and the main criticisms people have would be very helpful.

Costs/benefits of different kinds of third party model access

  • What this is:
  • Looking at the costs and benefits of different kinds of third-party access to front-end models. I'm using access very broadly to mean both pre-deployment and post-deployment, access for evaluations and also access for information about, for example, the training or the safety mitigations. By the benefits I mostly have in mind the benefits to safety, security, reduced P(bad outcomes); by the costs, I mostly have in mind things like security costs and how difficult it would be to get this concession from labs.
  • Why:
  • We probably want some kind of third party access, but right now there's nothing clearly laying out the benefits you'd get in terms of safety assurances from different kinds of access, or the costs in terms of security of different kinds.
  • Having paper(s) on this would help both advocate for particular kinds of access, and also the step zero of figuring out the kinds of access we should even be trying to get.
  • Who could do it:
  •  Probably a good fit for a summer research project, if supervised by someone who's thought a bit about third party access.
  •  The most ambitious versions of this probably benefit from some conversations with people working at frontier labs.
  •  There are purely technocratic versions of this, where you can just crunch a bunch of numbers about the different kinds of access and ways you can get access, it'd be very theoretical. And there are more policy-flavoured approaches where you advocate for a particular kind of access by first clearly laying out the benefits and the tradeoffs.
  • Am I thinking of doing this: I’m interested in supervising/mentoring work on this topic.

Bad cop METR

  • What this is:
  • Third party “how to do evals” org which is more adversarial towards labs. (Calls them out on bad practices, etc).
  • Adjacent to the model card scoring Luca Righetti and Tegan McCaslin have been working.
  • Why:
  • Who could do it:
  • You should have done some thinking about how to run evals. I don’t think you need to have literally designed/run evals, though.
  • Good public communicator.
  • Veneer of scientific rigor would probably help.
  • Am I thinking of doing this: no

Cheap asks for labs (transparency, safety)

  • What this is:
  • Assume we probably can’t make labs do stuff. What stuff is relatively cheap for them to do and very useful (for reducing P(bad outcomes))?
  • Why:
  • Seems like a useful frame.
  • Daniel K is into it.
  • Already quite a lot of effort here, relatively speaking, but unlikely we’ve totally exhausted the possibility space.
  • Who could do it:
  • People who like thinking about stuff and can have chats with lab(-adjacent) people.
  • Ideal profile has the clout to make these asks publicly, e.g. DK on model spec stuff.

Better threat modelling for evals (tied to particular actors and/or OCs)

  • What this is:
  • Rather than building an eval for generically interesting and/or scary things, do some fairly specific threat modelling, then build the eval.
  • Or, look at a big eval and figure out how it maps to certain actors/operational capabilities.
  • Why:
  • Makes red lines easier to generate
  • Generally seems good for preventing frog boiling
  • Who could do it:
  • Probably you should either be a scope-sensitive domain expert or good at BOTECy, shleppy research.
  • Luca’s team sorta doing this? Or like, doing the threat modelling bit.

Questions about digital minds and AI welfare

  • What this is:
  • Miscellaneous set of questions I’m interested in/confused about, didn’t think about for >10min
  • Why:
  • Digital minds could be a big deal
  • Who could do it:
  • Philosophers
  • People who like interacting with LLMs
  • People who enjoy somewhat abstract research

Anti-coup work

  • What this is:
  • Following up on Tom’s work, but trying to come up with concrete policy asks
  • Why:
  • Who could do it:

Philosophical work on AI

What this is:

  • Idiosyncratic set of stuff I think is kinda useful
  • Agency-only accounts of moral patienthood/the plausibility of these accounts
  • System prompts for LLMs
  • Thinking about the likely characteristics of future AI systems and how these bear on power-seeking/LoC arguments
  • Why:
  • [todo]
  • Who could do it:

Spending effort on elicitation/scaffolding to get SoTA performance from LLMs in some domains

  • What this is:
  • What it says
  • Why:
  • Feel the AGI
  • Demonstrate how easy it is to underelicit
  • Who could do it:
  • Tech bros with a spare month
  • The scarier versions are more important but would need higher integrity/discretion people to do them

The AI agent company

  • What this is:
  • Eli’s idea of a company run (transparently) by AI agents.
  • Why:
  • Who could do it:
  • Am I thinking of doing this: no.

Bounties for beating evals high scores

  • What this is:
  • A way of getting other people to do this.
  • Probably can only do this publicly for fairly non-scary capabilities.
  • Why:
  • Who could do it:
  • Am I thinking of doing this: no.

Consultancy for LLM augmentation/automation        

  • What this is:
  • Why:
  • Who could do it:
  • Am I thinking of doing this: no.

Side channel attack demos

  • What this is:
  • [todo]
  • Why:
  • Who could do it:
  • Am I thinking of doing this: no.

Figuring out this whole “AI labour” thing

  • What this is:
  • Why:
  • Who could do it:
  • ?
  • Am I thinking of doing this: no

Scoping a bunch of def/acc stuff for startuppy types to work on

  • What this is:
  • Why:
  • CE handed founders projects on a plate and that seemed to go great
  • Funding appetite
  • Who could do it:
  • Am I thinking of doing this: no

Red-teaming evals best practices

  • What this is:
  • [todo]
  • Why:
  • Who could do it:
  • Could be a summer fellowship
  • Am I thinking of doing this: no

The 10-page explainer of LoC for a normal audience, with evidence and stuff

  • What this is:
  • Why:
  • Who could do it:
  • Am I thinking of doing this: no.

Producing evidence of LoC

  • What this is:
  • Why:
  • Who could do it:
  • Am I thinking of doing this: no.

Scary demos

  • What this is:
  • [todo]
  • Probably lots of overlap w/ other lists
  • Why:
  • Who could do it:
  • Am I thinking of doing this: no.
  • ITN score:  *  *  = /125

Accelerating philosophical progress with AI

  • What this is:
  • Do stuff to make LLMs better at conceptual non-verifiable reasoning, where “stuff” includes schlep and/or fun ML
  • E.g. reasoning traces, debate as ground truth, inferring later positions from earlier ones (perhaps in fictional philosophical systems), mapping the space of internally consistent positions, better scaffolding for abductive reasoning
  • Why:
  • Solving philosophy is probably pretty important and won’t be solved by default.
  • Transfers to conceptual alignment research?
  • Who could do it:
  • Am I thinking of doing this: no

IDs for AI agents/human users

  • What this is:
  • Proof of concept actual product for this
  • Why:
  • Who could do it:
  • Startup bros, tech bros, def/accers,
  • Am I thinking of doing this: no, could put you in touch with the proof of personhood crowd though

Template

  • What this is:
  • Why:
  • Who could do it:
  • Am I thinking of doing this: no.
  • ITN score:  *  *  = /125