How to Read a Research Paper^[1]

Dmitry Berenson

Introduction

Understanding a research paper is like creating a painting. Because paint needs to dry before more paint is applied on top of it, paintings are created in layers. For instance, consider a landscape: First, the painter creates a green blob where the grass should be and a blue blob where the sky should be, a purple blob for a mountain, etc. Second, after the first layer dries, the painter puts in the shapes of the trees or streams and other large features, thus defining the composition without defining the specific details of the shapes. Third, the painter adds details overtop the shapes and blobs that define the leaves on the trees, the ripples in the stream, the touch of snow on the mountains, etc. When we look at a painting with an untrained eye, we are usually drawn to the top-most level; the level with the most interesting detail. Yet without the proper context, a detail is just a couple strokes of paint. It is the context of the previous layers that make a leaf out of a blotch of green or a ripple out of a white line.

This guide will teach you how to read a research paper in layers, as if “painting” the understanding into your mind. Reading a paper in this way will require that you re-read certain sections of the paper twice or even three times. This may seem redundant, but keep in mind that each time you read a section you’re looking for different things. For instance, the first time you will skim over most of the implementation details in a section, but in the third and fourth times you will focus on them.

This guide presents a 4-level approach to read a research paper. The levels are ordered from broad (1) to specific (4). To apply the method presented here, you must first read the paper at all levels lower than the level you’re going for. So to read it at level 3, you must read it at level 1, then at level 2, then at level 3. Attempting to read the paper at level 3 or 4 is a waste of time unless you understand levels 1 and 2, you will just get confused and irritated. Regardless of the level you choose, always end your reading by stepping back and assessing the paper as a whole (see the last section of this guide).

Before you begin

Before reading a single word of the paper, you have to be clear on why you’re reading it in the first place. Is it for a literature review? A potential citation for your paper? A class presentation? The reason to clarify your purpose for reading the paper is to determine which level you should aim for. I’ve included a table with some of the most common purposes below:

Keep in mind that this is the intention you set before reading the paper. Once you start reading, you may become interested and want to go deeper. If so, go for it!

Level 1: The Main Idea

The purpose of this level is to answer the following questions:

1. What type of problem/phenomenon are the authors trying to solve/investigate?
2. Why should the reader care about this problem/phenomenon?
3. How is this approach different from previous work?
4. Why is this approach better than previous work? (more general, faster, etc)
5. What type of approach is this? (there are usually several types of approaches for solving the same problem)
6. What are the applications of this work?

The answers to most of these questions will be found in the abstract, introduction, related work, and problem statement. You can also take a look at the conclusion. In engineering contexts, I also recommend taking a look at the Results section. This section will show you what actual problems the authors have applied their method to. Often these problems are far simpler than the ambitious language of the abstract and introduction might suggest. If this is the case, it is then up to you to determine if the paper is worth reading, or if the method seems to only work on toy problems. Keep in mind that if the method did indeed work on real-world problems, the authors would have included this in the paper.

Level 2: The Structure

You must read the paper at level 1 before continuing to this level.

The purpose of this level is to build a block-diagram of the components of the method. I recommend drawing this diagram on paper. Each block should have a name and inputs and outputs. Sometimes the authors will include such a diagram in the paper, sometimes they won’t. Even if they include it, you have to understand what each block does and how it interfaces with the other blocks.

In many cases, the block-diagram will seem very straightforward, i.e. do step 1, then do step 2, etc. Don’t be fooled! A single step can hide a lot of sub-steps that are critical to understanding the paper. Make sure you catch each of these important sub-steps.

Though you may be tempted, now is not the time to dig into the details of each step or sub-step! Your goal at this level is to identify the key components of the method and how they work together. Pay special attention to the inputs and outputs of each step and make sure you understand what each one is!

After reading at level 2, you should be clear on the structure but you may have no idea how each block works. That’s OK because

1. Usually most of the blocks are mundane and you don’t really care how they work, you just care about their inputs and outputs
2. Some blocks are not relevant to the main thrust of the paper, or are not relevant to your interest
3. You will drill down into the block(s) you care about in level 3

Level 3: The Most Important Details

You must read the paper at levels 1 and 2 before continuing to this level.

Now that you have the block diagram in place and you understand why the authors are doing what they’re doing, it’s time to dig into the details of the main contributions of the paper.

Usually the authors will be basing their contribution on previous work, so track down any citations you don’t know about that are central to the author’s method. Sometimes the authors will just mention these in passing, such as “and we use the ____ algorithm [4] to compute the ____ property of the dataset.” Even though the authors don’t spend much time discussing how they use the other work, it may be essential that you understand how the cited work is applied because subsequent steps probably rely on it. This means you should find the cited work and read through it at at least level 1, but possibly at level 2 if it’s really important.

Now you need to pay very close attention to the author’s contribution and make sure you understand each step of what they’re talking about. A useful exercise is to imagine a given input and see how the method would handle that input. For instance, for the A* algorithm, imagine a maze environment with a start and goal and draw out the cells the algorithm would expand as it searched (you don’t need to draw all of them, just enough to get the intuition). You might even imagine a difficult case for the method (for instance a cul-de-sac for A*) and think about how the method would cope with it.

This is also the time to identify anything you don’t understand in the math or notation of the paper. Don’t let this slow you down when you first read through the section describing the method, just mark what seems confusing. Then, when you’re done with the first read go back and figure out all those confusing parts.

In engineering contexts, this is also the right level to take a close look at the results section. The following questions are helpful to think about when reading the results section:

• Do the results justify the method?
• Are there cases/scenarios that should have been tested but weren’t?
• Is the complexity of the method justified (in terms of some quantifiable metric) by its advantages over other methods that may be simpler or easier to use?
• Where does the method fail?

Level 4: The Other Details

You must read the paper at levels 1, 2, and 3 before continuing to this level.

At this level, you are really looking into every aspect of the method in great detail. You may be doing this because you are trying to implement the method or looking for subtle differences from your own work. I don’t recommend this level unless you really need to know the paper inside out.

Now it’s time to look at all the other boxes in your block diagram that are not the main contribution and investigate how to implement them. You may also be looking for subtle details in the implementation of the method that cause it to perform poorly for a case you’re interested in so that you can improve the method.

Unfortunately, at this point, there is little general advice I can give, other than 1) be thorough and 2) if you’re trying to implement the method, you will probably get it wrong the first few times. Don’t worry, getting it wrong is natural because the authors have inevitably left something out of their implementation description. Part of your task will be to fill in the blanks as best you can.

Stepping Back

Now that you’ve read the paper at the appropriate level for your goal, it’s time to step back and appreciate the overall picture. At this point, it’s useful to ask these key questions about the paper:

1. What are the strengths of the paper, i.e. what does it do well?

The authors will shout the strengths of their paper at maximum volume but don’t take their word for it. Do the results really justify their claims?

2. What assumptions does the paper make?

Sometimes the assumptions are stated very clearly, but usually there are some hidden assumptions as well. These assumptions can limit the method’s applicability to some important cases or can make the method unusable in real-world situations. A good way to discover the hidden assumptions is to try to think of an example which “breaks” the method.

3. How can this work be improved?

While it’s possible the method could be improved by shedding assumptions, some assumptions are so fundamental to the method that they cannot be removed unless you change everything. Asking what can be improved is essentially asking what the next step would be if you were to continue making the fundamental assumptions that this method makes.

You might think that the next steps are told to you in the future work or discussion sections, and indeed some next steps are usually suggested there. Unfortunately, these suggestions are often afterthoughts and not really worth pursuing. Try to be creative and think of improvements that the authors haven’t suggested.

The level at which you read the paper will dictate how thoroughly you can answer the above questions. However, even at level 1, you should be able to come up with rough ideas for all of them. When you read at higher levels, you will be able to answer these questions more fully.

After answering these questions, it’s time to assess your understanding with respect to your goal for reading the paper. For instance, if you’re reviewing the paper, do you understand enough of it to make a fair judgement? For presenting to a class, can you describe the motivation and block diagram, as well as the main contribution? If you find that you can’t do what you set out to do, it may be time to seek help or to read more background literature before re-reading the paper.