Pre-read: common pitfalls

Limits to Prediction (Spring 2024)

Arvind Narayanan

We will continue with the theme that building models is the relatively easier part of ML and evaluating their performance is the harder part. Last time we discussed performance metrics. This time we will learn about the wide variety of ways in which researchers fool themselves into overoptimistic performance estimates, and try to get a sense of how often this affects the published literature.

The first paper started life as a class project in the previous edition of this class. Finding leakage in some of the papers on the reading list clued us into the fact that this is a much more widespread problem. (The link goes to a webpage that gives additional context that might be helpful.)

Next up is REFORMS, a paper by a large team of collaborators we assembled after the leakage work. It proposes a checklist to address many possible errors in ML-based science (of which leakage is just one). Our main interest in this class is not the checklist, but rather the paper’s overview of all the ways in which things can go wrong, grouped into eight main categories.

The next paper is a good illustration of so-called “researcher degrees of freedom”, closely related to the well-known problem of p-hacking in the data modeling culture, but equally problematic in the algorithmic modeling culture. The paper is broader than researcher degrees of freedom; it will also be useful as a reminder of some topics previously discussed in the course such as prediction vs explanation and skill vs luck.

The above papers are about ML-based science, which is about applying ML to make claims about predictability of life outcomes or weather or other phenomena. Overoptimism also besets research on developing new ML methods, which makes claims about algorithms or models rather than populations. But the problems manifest in somewhat different ways in the latter community. A good overview is the paper Troubling Trends in Machine Learning Scholarship, particularly interesting because it connects the observed troubling practices to the culture of the ML methods community.

Things have gotten better in some ways since then, and worse in other ways. With LLMs, a notable evaluation pitfall is contamination, which has led many to argue that the whole evaluation paradigm is breaking down.

For Wednesday’s class, we’ll talk about ways to avoid some of these pitfalls.