1 of 35

Pre-registration course work on research and Presentation, VNSGU Surat�

Selective reporting and misinterpretation of data

Dr Ranjitsinh Devkar PhD

Metabolic Endocrinology and Chronobiology lab, Department of Zoology

The M.S. University of Baroda, Vadodara

2 of 35

3 of 35

Spectrum of research practice�

How it should be done:

Relevant, Valid, Reproducible, Efficient

Sloppy science:

Ignorance, honest error or dubious integrity

Scientific fraud:

Fabrication, Falsification, Plagiarism

3

Responsible

Conduct of

Research

Questionable

Research

Practices

Research

Misconduct

4 of 35

4

DETERMINANTS OF BAD PRACTICES

SYSTEM

publication pressure

hyper competition

low risk – high rewards

CULTURE

wrong role models

insufficient mentoring

no RCR education

no clear guidance

INDIVIDUAL

justifying misbehavior

conflicts of interest

moral attitudes

personality traits

In fact we know very little about the determinants of QRS and RM

This slide just summarizes some plausible ideas

The determinants at the system level are often referred to as perverse incentives, while at the individual level the term bad apples is popular

It seems likely that determinants from these three catagories interact

Which would be good news, because a focus on factors with a central role in the causal network would have a strong preventive effect

We urgently need research to provide the evidence-base for effective interventions

Examples of research questions that need to be answered are:

Will changing the incentives at the system level lead to the desired changes in the research culture, and will that attract individuals who are more inclined to RCR?
Should we focus on RCR education and empowerment of PhD students and postdocs, or should senior staff get most attention?
Are quality sytems and internal audits effective in preventing QRP and RM?

Joeri Tijdink will present his study on publication pressure among Dutch medical professors during this conference

5 of 35

5

Ranking research misbehavior

60 items ranked by 34/59 experts

How often will this misbehavior occur?

very rarely (1) – rarely (2)– regularly (3) - often (4) - very often (5)

If it occurs, how large will its impact be on the

validity of knowledge?

negligible (1) – small (2) – medium (3) - large (4) - enormous (5)

6 of 35

7 of 35

8 of 35

Why does selective reporting or misinterpretation of data occur??

9 of 35

10 of 35

11 of 35

12 of 35

Linear decrement in data: Always not true

13 of 35

What do you think about this data??

14 of 35

15 of 35

16 of 35

16

average of 21 surveys

Self-reported Falsification or Fabrication at least once in last 3 yrs 🡪 2%

Self-reported Questionable Research Practice at least once in last 3 yrs 🡪 34%

Research misconduct and questionable research practices occur

These figures come from the highly cited meta-analysis of Daniele Fanelli

A recent survey shows higher numbers for research misbehavior in economics (4% and 94%) 🡪 more honest reporting and/or more misbehavior?

Self-reported 🡪 Conservative estimates of true prevalences

Difficult to translate to % papers with both categories of misconduct 🡪 but you will agree that retractions show only a very tiny tip of the iceberg

Self report of fabrication and falsification implies intention to deceive

Self report of QRP and non-self reports concern uncertain intention to deceive

Fanelli D. How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data. PLoS ONE 2009; 4(5): e5738

Necker S. Scientific misbehavior in economics - Research Policy 2014; 43: 1747-59

17 of 35

17

conflicts

of

interest

sponsor

interests

QRP & RM

(false)

positive

results

citations

publications

media

attention

grants

&

tenure

HOW THINGS CAN GO WRONG

This slide shows – as a simplified interplay of determinants – how things can go wrong leading to selective publication

In most disciplines the proportion of papers reporting positive results has increased over time

Positive results are published and cited more often, and also get more media attention

This will probably increase the likelyhood of getting grants and tenure

We have also some evidence that confilcts of interest and sponsor interests may lead to false positive results, because individuals and sponsors have also other interests than truth finding

This mechanism will lead to publication bias, selective reporting and selective citation

These phenomena will distort the truth in the published record and can explain the large replication difficulties that many disciplines experience

There is strong evidence for some of the relations suggested in this slide, but no or only little evidence for most of them

We really need more solid empirical research to clarify how these things work

Gaining this knowledge is important for effectively fostering RCR and preventing QRP and RM

Dwan K et al. Systematic review of the empirical evidence of study publication bias and outcome reporting bias. PLoS ONE 2008; 3(8): e3081.

Knottnerus JA, Tugwell P. Selection-related bias, an ongoing concern in doing and publishing research. J Clin Epidemiol 2014; 67 1057-8.

van Assen MA, van Aert RC, Nuijten MB, Wicherts JM. Why publishing everything is more effective than selective publishing of statistically significant results. PloS one. 2014;9(1):e84896.

Fanelli D. Negative results are disappearing from most disciplines and countries. Scientometrics 2012; 90: 891-04.

18 of 35

19 of 35

19

Non-publication 🡪 publication bias

Selective reporting 🡪 reporting bias

Both favour preferred (‘positive’) findings
Leading to a distorted picture in the published body of evidence

🡪 Flawed Systematic Reviews

🡪 Low Replication Rates

20 of 35

20

21 of 35

21

Only 6 of 53 preclinical landmark cancer studies

could be confirmed by replication

When negative studies are rarely published,

published positive studies are likely to be chance findings

Non-confirmed studies

sometimes inspire many new studies 🡪 research waste!
sometimes lead to clinical trials 🡪 unethical situation!

22 of 35

Avoidable waste may be up to 85%

22

23 of 35

23

Prevention of selective reporting of clinical trials

Registration + uploading of protocols, data and publications

Quality of reporting

www.equator-network.org

N = 270

In the field of randomized clinical trials we have seen some progress in during the last two decades

Prior registration of RCTs became the norm and increasingly these registries enabled or even demanded uploading of protocols, data-sets and publication

Also many reporting guidelines became avilable

Funding agencies and journals made these good ideas increasingly an obligation

Gerben ter Riet will present an example of a system to prevent selective reporting during his presentation during this conference

Ter Riet G, Bouter LM. How to end selective reporting in animal research. In: Animal models in research and development of cancer therapy (in press).

Goldacre B. Are clinical trial data shared sufficiently today? No. BMJ 2013; 347: f1880.

Chalmers I, Glasziou P, Godlee F. All trials must be registered and the results published. BMJ 2013; 346: f105.

Hudson K. Sharing results of RCTs. JAMA published online November 19, 2014 E1-2

Swaen GM, Carmichael N, Doe J. Strengthening the reliability and credibility of observational epidemiology studies by creating an Observational Studies Register. J Clin Epidemiol 2011; 64: 481-6.

24 of 35

24

www.alltrials.net

25 of 35

25

The sad news

Slow rate of adoption (in clinical research)

50% of registered data is not published
50% of published data is not registered
Open Data is slowly gaining momentum

Room for improvement

An inspiring example for other disciplinary fields

26 of 35

26

Identification

of

27 of 35

27

Conditions for transparency

adequate skills, systems and facilities
some months of embargo
proper acknowledgements
opportunity to participate
guarantees against breaches of privacy and misuse
predefined study protocol for re-use of data

I firmly belief that data collected among volunteering participants of (clinical) research belong to the public domain – and also that this is an important principle outside of clinical research

Of course, some months of embargo can be reasonable, proper acknowledgements should be made, and maybe the original investigators should be offered the opportunity to participate in the secondary analyses

Breaches of privacy and misuse of the data ought to be prevented, and also obstruction and intimidation by excessive open records requests

Open record laws can have nasty side effects – more uniform disclosure standards are needed

Data sharing clearly is the norm, but how to do this is less obvious – although some very promising projects are underway

And it is obvious that for secondary analyses the same rules for transparency apply, starting with a predefined study protocol.

Halpern, Mann - Open records versus harassment - Science 2015; 348 479

IOM (Institute of Medicine). 2014. Discussion framework for clinical trial data sharing: Guiding principles, elements, and activities. Washington, DC: The National Academies Press

28 of 35

28

How can we promote transparency?

re-design reward system

Prestige and tenure depend on publications, citations and grants
Reward publication of protocols and ‘negative’ results
And reward data sharing and replication

29 of 35

30 of 35

Lack of global or international exposure may affect interpretation of data.

31 of 35

32 of 35

33 of 35

33

Conclusions

Sloppy science is a larger evil than research misconduct
Especially selective reporting threatens validity and efficiency
More transparency is urgently needed
Factors in system, culture and individual are ‘holding us back’
We must change the reward system and face our dilemmas

34 of 35

35 of 35

Sources