How to Review a Building Simulation Paper

Nathaniel Jones, PhD, Arup

February 3, 2022

Research Committee Webinar

Objectives

1. Understand the role that peer review plays in IBPSA conferences
2. Know what to look for in a scientific paper
3. Describe the components of a useful review
4. Identify when it is appropriate to reject a manuscript

Who Am I?

Nathaniel Jones, PhD

• Senior consultant at Arup
• BPACS 2022 scientific committee member
• 50 completed journal peer reviews

Who Are You?

• It’s my first time reviewing a paper
• I’ve reviewed a few scientific papers before
• I’m an experienced peer reviewer
• None of these. I’m just curious

Purpose of Peer Review

• Regulate what becomes part of the body of scientific literature
• Give feedback to help the authors improve the paper as much as they can
• Help the authors to become better in future papers

Flickr/AJ Cann, CC BY-SA

Myth vs Truth

The Big Question

Is it relevant?

• The topic is related to building performance simulation
• People at the conference will be interested in this topic

Is it significant?

• The research question is worthy of investigation
• The research findings could advance the state of the art

Is it novel?

• The method has never been done before
• The researchers have an original approach

Does this paper contribute to the field?

Parts of a Paper

Why did the authors feel the need to write this paper?

• Is it easy to understand?
• Does it require the reader to already be familiar with the subject matter?
• Look for the statement of contribution:�This paper…

Introduction

Parts of a Paper

What do you need to know to understand this paper?

• Is there sufficient background information?
• Are peer reviewed papers from a variety of authors cited?
• Did the authors try to explain the entire field of _______ in one page?

Background

Parts of a Paper

What did the authors do?

• Is the process repeatable?
• Is the process dependable?
• Is the process new?

Method

Parts of a Paper

What happened?

• Are the results presented clearly?
• Are the results surprising?
• If they are completely unexpected or improbable, could there be a mistake?
• If they are completely unsurprising, is there any reason to publish them?

Results

Parts of a Paper

Why should we care?

• Is the interpretation of the results interesting?
• Are the relevance, significance, and novelty of the paper clear?
• Are there recommendations for:
• The reader (actions to take)?
• The authors (future work)?

Discussion

&

Conclusion

Accept or Reject?

“The benchmark for acceptance is whether the manuscript makes a useful contribution to the knowledge base or understanding of the subject matter”

– Wiley

• Accept as is
• Accept with minor revisions
• Review again after major revisions
• Reject

}

This is your recommendation.

Do not assume the scientific committee will follow it.

Reasons for Revisions

1. Does the title properly reflect the subject of the paper?
2. Does the abstract provide an accessible summary?
3. Do the keywords accurately reflect the content?
4. Is the paper an appropriate length? Is it boring or repetitive?
5. Are the key messages short, accurate and clear?
6. Is it clear what the authors did (and why they did it)?
7. Does the literature review explain the state of the art prior to this paper?
8. Does the hypothesis or research question make sense?
9. Does the method make sense as a way to test the hypothesis?
10. Are the modeling assumptions and standards used appropriate?
11. Do the results make sense? Is there anything that points to a mistake in calculations or setup? If any results were unexpected, is a reasonable explanation provided?
12. Do the figures and tables match the text? If the authors say that a figure shows some result, is it easy to interpret the figure in the way the author did?
13. If there is statistical analysis, does it make sense? Are the statistical measures appropriate to the question being asked?
14. Are the conclusions supported by data from the paper?
15. Are the references well-respected and peer reviewed?

When to Reject

Do not assume the scientific committee will follow your advice.�You must still provide constructive feedback.

When to Present as Poster

Conference Attendance

Posters

When to Present without a Paper

Conference Attendance

Presenters

Who Makes the Cut?

• A case study of a famous building, but similar studies have been done for other buildings
• Interesting results, but difficult to understand because the paper is full of grammatical and spelling mistakes
• A novel method, but there are no results, just an extended hypothetical discussion
• A well written paper with surprising results, but every image has the author’s corporate logo on it

Setting Up Your Review Space

How people think I review papers

How I actually review papers

© Dennis Wilkinson

© Mack Male

If You Prefer Printed Pages

• Leave a margin to write in
• Use multiple pen colors
• Write neatly
• Check off or highlight comments when you transfer them to your typed review
• Not every comment needs to be included in the final version

If You Prefer Screens

• Keep the manuscript PDF and your notes side-by-side
• Enter notes with reference to page, paragraph, or line number as you read
• Push important comments to the top
• Review your notes after reading through the manuscript – your comments might have been addressed later on

Anatomy of a Review

First Paragraph

1. Summarize the paper’s contribution in one sentence
2. Summarize the strengths and weaknesses of the paper
3. Make the case for (or against) including this paper in the conference.

This paragraph shows the author how well their writing was understood, and it may be the only part the scientific committee reads.

The paper describes a new approach to modeling ground source heat pumps using artificial neural networks. The example case study is well documented, but the description of the neural network itself is too vague to allow reproducibility. The innovative use of neural networks will gain a lot of attention at the conference if the authors can revise it to make their method clearer.

Anatomy of a Review

Major Issues

1. Identify problems with the research question, method, and analysis of the results.
2. Give advice.

“Offer clear suggestions for how the authors can address the concerns raised. In other words, if you're going to raise a problem, provide a solution.”

– Jonathon Halbesleben, Editor of Journal of Occupational and Organizational Psychology

• Number the issues for easy reference.

1. The baseline efficiencies are unrealistic. Please cite evidence of these efficiencies or use a CoP in the range 3 to 4.5.
2. Please provide details about the artificial neural network, including the software used, number of layers, and source of the training data.

Anatomy of a Review

Minor Issues

1. Identify simple mistakes that can easily be corrected, such as typos, formatting errors, mistakes in figures and tables, and misstatements or misinformation.
2. It’s not your job to point out every minor mistake, but don’t expect the authors to find mistakes on their own.
3. Spelling, grammar, and punctuation:

“If a manuscript has many English language and editing issues, please do not try and fix it. If it is too bad, note that in your review and it should be up to the authors to have the manuscript edited.”

– Wiley

Page 2: Please enlarge the text in Figure 1 to make it legible.

Page 3: The meaning of the 3^rd paragraph is unclear. Please revise.

Page 4: The values in Table 2 do not match those given in the text. Please correct any typos.

Overall: The grammar is poor. Please consult your university’s writing center.

Anatomy of a Review

Last Paragraph

1. Summarize strengths and weaknesses of the paper.
2. Summarize actions needed from authors.
3. If not a good fit, recommend a venue that would be a better fit.

Overall, this paper makes a clear argument for the utility of neural networks for heat pump design, but it does not meet the level of technical rigor expected for this conference. In the next revision, please expand the method section with more technical explanation. Otherwise, Conference B might be a better fit for this paper.

What Would You Say?

… This paper presents survey data from 8 building occupants proving that sunlight does not cause glare on computer screens. Our findings refute previous work by Smith which suggested that sunlight could cause glare...

• A sample size of 8 surveys is too small for this study
• Words like “proving” and “refute” should be replaced with “suggesting” and “contradict”
• There may be a problem with the method, as the results seem unlikely

​

What Would You Say?

… Our findings are shown in Figure 8...

​

​

​

​

​

Figure 8: The results

• The axes of the graph need to be labeled
• The caption of Figure 8 does not give enough information to understand the graph
• Please provide a statistical analysis of the results

​

What Would You Say?

… Our findings show a positive correlation (Figure 8)...

​

​

​

​

​

Figure 8: Dependence of traffic ticket cost on automobile speed

Rhys Baker, How to Draw a Scientific Graph: A Step-by-Step Guide

Your Job

• Inform the scientific committee of the relevance, significance, and novelty of each paper
• Give the authors feedback to help them get their message across clearly

​

​

Good luck in your reviews!

Nathaniel Jones, PhD

nathaniel.jones@arup.com

​

To add…

• Pros and cons of peer review
• Conflicts of interest / reasons to refuse to review