1 of 59

Chapter 4��Reading and Critiquing Research Reports

2 of 59

Types of Research Reports

Theses, dissertations, books
Presentations at professional conferences

Oral reports: 10-20 min to describe the key features of study
Poster sessions: visual displays summarizing the study

Journal articles

Papers often subjected to peer review (2 or more)
Peer reviews are often blind (reviewers are not told names of authors and vice versa)
Acceptance rate 5%

3 of 59

Content of Research Journal Articles

IMRAD Format:

Title and Abstract

Introduction
Method
Results
And Discussion

References

4 of 59

The title and abstract:

The title conveys the nature of the study in 15 or fewer words.

In quantitative studies it includes: variables and population
In qualitative studies, it includes: the central phenomenon and group under investigation.

5 of 59

Abstract

The abstract is a brief description of the study placed at the beginning of the article (100-150 words).
Brief description of major features of a study at the beginning of a journal article
Content are ………

Old style—single paragraph, about 100 to 150 words
New style (stuctured)—more detailed, with specific headings

6 of 59

Introduction

Description of:

Central phenomena, concepts, or variables
Study purpose, research questions, or hypotheses
Review of literature
Theoretical/conceptual framework
Study significance, need for study

7 of 59

Method Section

Methods to answer the research question
Quantitative Studies:

Research design
Sampling plan and characteristics of study participants
Methods of operationalizing variables and collecting data
Study procedures, including procedures to protect participants
Analytic methods and procedures

8 of 59

Method Section (cont.)

Qualitative Studies:

Research tradition
Sampling approach and description of study participants
Setting and context
Data collection approaches
Study procedures
Analytic strategies

9 of 59

Question

Is the following statement True or False?

The review of literature is typically found in the Method section of the research report.

10 of 59

Answer

False

The review of literature is usually found in the Introduction of the research report.

11 of 59

Results Section

Findings:

Quantitative Studies:

Descriptive information (e.g., description of subjects, key veriables)
Results of statistical analyses

Names of statistical tests (prcedures for testing hypotheses)
Value of calculated statistics

Level of statistical significance

Significance means: findings are true and replicable

Precision and magnitude of effects: precision of effect and effect size (important finding or not)

12 of 59

Results Section (cont.)

Findings (cont.)

Quantitative Studies (cont.):

Level of statistical significance—index of how probable it is that the findings are reliable
For example, p < .05: Probability is less than 5 in 100 that the findings are spurious (probability is 95 in 100 that the findings are “real” and replicable)

13 of 59

Results Section (cont.)

Findings (cont.)

Qualitative Studies:

Findings often organized according to major themes, processes, or categories identified in the analysis
Almost always includes raw data—quotes directly from study participants

14 of 59

Discussion

Interpretation of the results
Implications for nursing practice and for further research
Study limitations

15 of 59

Question

Is the following statement True or False?

A researcher would describe the limitations of the study in the Results section of the report.

16 of 59

Answer

False

The study limitations are typically described in the Discussion section of the report.

17 of 59

Style of Research Journal Articles

Why are research articles so hard to read?
Often difficult to glean the “story” being told, because of:

Compactness—page constraints
Jargon: research terms
Objectivity, impersonality (use passive voice)
Statistical information

Last two especially prominent in quantitative research articles

18 of 59

Tips on Digesting Research Reports

Read regularly, get used to style
Read copied articles—underline, highlight, write notes
Read slowly
Read actively: monitor yourself to determine whether you understand what you are reading.
Look up technical terms in glossary: text books glossary
Don’t be intimidated by statistics—grasp gist (general idea) of story
“Translate” articles or abstracts: to more familiar terms

19 of 59

Research Critiques

Careful and objective appraisals of the strengths and limitations of a study
Critiques of individual studies can be done for a variety of reasons (e.g., for a student assignment; for making decisions about whether or not to publish a manuscript; for EBP purposes)
Vary in scope, length, and form, depending on purpose

20 of 59

Research Critiques (cont.)

Can be comprehensive, appraising the:

substantive: significant and important
Methodologic: rigorous, appropriate, the findings sound
Theoretical: the framework sound
Ethical: participants rights
Interpretive: appropriate interpretations and defensible conclusions
and stylistic aspects of both the study and the report (e.g., students can critique a single study to demonstrate their research skills): clear correct and organized

Noncomprehensive critiques tend to focus on key substantive and methodologic issues, focusing on the integrity of the study’s evidence.

21 of 59

Research Critiques (cont.)

Critiques can be facilitated by:

Using a formal protocol or critiquing guideline—although a one-size-fits-all guideline does not typically work perfectly (see guidelines for quantitative and qualitative studies in this chapter as models)
Reviewing a model of a good critique

22 of 59

Title

Good one
Suggesting the key variables and population

23 of 59

Abstract

Clear
Concise
Summaries the main feature of the report

24 of 59

Statement of the problem

Easy to identify
Persuasive
Significant: important
Method are appropriate to the problem

25 of 59

Hypothesis or research questions

Explicitly states; if not why?
Appropriately worded; clear variables; population present
Consistent with literature review and conceptual framework

26 of 59

Literature review

Up-to-date
Primary source
State-of-the-art synthesis of evidence
Solid bases for a new study

27 of 59

Conceptual / theoretical framework

Conceptual definition is adequate
Rational for it and it is appropriate
If absent: is it justified?

28 of 59

Protection of participants rights

Appropriate procedures
IRB
Risk/benefit ratio

29 of 59

Research design

Most rigorous used
Appropriate comparisons made to enhance interpretability of the findings
Appropriate data collection points
Minimize biases

30 of 59

Population and sample

Population identified
Sufficient description of the sample
Best possible sampling design
Sample biases decreased
Adequate sample size
Power analysis used

31 of 59

Data collection and measurement

Operation and conceptual definition were congruent
Key variable operationalized using the best method with justification
Full description of the instrument and was it appropriate
Valid
Reliable

32 of 59

Procedures

Adequate description of the intervention
Appropriate implementation
All the subjects received the intervention
Minimize bias
Staff training

33 of 59

Data analysis

Done for each question and hypothesis
Appropriate statistical method
Most powerful method used
Minimize type I and type II error

34 of 59

Findings

Statistical significance mentioned
Effect size and Precision
Adequate summary of the findings
Good use of figures and tables
Sufficient for meta-analysis and EBP

35 of 59

Interpretation of the findings

All major findings interpreted within the context of previous studies and conceptual framework
Justification for casual relationships
Consistent with the results and limitation
Generalizability was addressed

36 of 59

Implication and recommendation

For clinical
Further research
Was it reasonable and complete

37 of 59

Presentation

Well written
Well organised
Written in a manner that is accessible for nurses
CONSORT flaw chart for participants was used

38 of 59

Researcher credibility

Qualified
Has experience

39 of 59

Summary assessment

Valid findings despite the limitations
The study has meaningful evidence that can be used in nursing practice

40 of 59

Key Challenges of Doing Research

Researchers face numerous challenges and make many decisions:

Conceptual definition
Financial
Ethical
Practical
Methodological

41 of 59

Question

Is the following statement True or False?�

Research critiques are done to evaluate the strengths and limitations of a study.

42 of 59

Answer

True

Although there are a wide variety of reasons for critiquing research, research critiques provide careful and objective appraisals of the strengths and limitations of a study.

43 of 59

Major Methodologic Challenge

Designing studies to support inferences (conclusions) that are:

Reliable and valid (quantitative studies)
Trustworthy (qualitative studies): the participants experience not the researcher perceptions

44 of 59

Criteria for Evaluating Quantitative Research

Reliability

The accuracy and consistency of obtained information

Validity

The soundness of the evidence—whether findings are convincing, well-grounded, and support the desired inferences

45 of 59

Evaluative Criteria in Qualitative Studies

Trustworthiness

The overall integrity of the study’s evidence

46 of 59

Trustworthiness

Dimensions of trustworthiness:

Credibility—a key criterion, achieved to the extent that researchers can engender confidence in the truth of the data and their interpretations
Confirmability: objectivity of data
Dependability: stability of data over time
Transferability: to new setting
Authenticity: show a range of different realities and transfer the feeling tone of lives as they are lived

47 of 59

Triangulation

Triangulation is the use of multiple sources or referents to draw conclusions about what constitutes the truth.

Triangulation can contribute to credibility.
Triangulation is a useful strategy in both qualitative and quantitative research.

48 of 59

Question

Which of the following would be a key aspect of trustworthiness?

Triangulation
Reliability
Credibility
Validity

49 of 59

Answer

c. Credibility

Credibility is a key criterion for trustworthiness. Reliability and validity are key for evaluating quantitative research. Triangulation is the use of multiple sources or referents to draw conclusions.

50 of 59

Bias

An influence producing an error and distortion in study results
Examples of factors creating bias:

Lack of participants’ candor (honesty)
Faulty methods of data collection
Researcher’s preconceptions
Participants’ awareness of being in a special study
Faulty study design

51 of 59

Research Control

In quantitative studies, research control involves holding constant extraneous factors (confounding variables) that influence the dependent variable, to better understand relationships between the independent and dependent variables.

Research control is one method of addressing bias.

52 of 59

Randomness

Randomness—allowing certain aspects of the study to be left to chance rather than to researcher or participant choice
An important tool for achieving control over

confounding variables
and for avoiding bias

53 of 59

Masking (Blinding)

Masking—or blinding—involves concealing information (usually about the study hypotheses or about participants’ status in different groups) from those playing a role in the study.

Used in quantitative studies to reduce biases stemming from awareness

54 of 59

Masking (Blinding), cont.

Single blind—only one group is blinded (e.g., study participants)

Double blind—two groups (e.g., study participants and people collecting the outcome data) are blinded

A study without blinding is sometimes called an open study, while a study with blinding may be referred to as a closed study.

55 of 59

Reflexivity

To control personal bias
The process of

reflecting critically on the self,
and of attending to personal values that could affect data collection and interpretations of the data

A strategy used primarily by qualitative researchers (e.g., they may maintain an ongoing journal to record their reflections before and during the study)

56 of 59

Question

Is the following statement True or False?�

Blinding is used in qualitative studies.

57 of 59

Answer

False.

Blinding is used in quantitative studies to reduce bias that results from awareness.

58 of 59

Generalizability and Transferability

Generalizability (Quantitative research): The extent to which study findings are valid for others not in the study
Transferability (Qualitative research): The extent to which qualitative findings can be transferred to other settings
Thick description (Qualitative research): thorough description of the setting and process, it supports transferability.

59 of 59

End of Presentation