1 of 32

Investigating pandemic drugs and vaccines

(Note: these methods are applicable outside a pandemic)

Peter Doshi, PhD

Senior editor, The BMJ Investigations Unit

Assoc. Prof, Univ. of Maryland School of Pharmacy

September 21, 2023

FINANCIAL DISCLOSURES

I have received travel funds from the European Respiratory Society (2012), Uppsala Monitoring Center (2018) and GIJN (2023); grants from the FDA (through University of Maryland M-CERSI; 2020), Laura and John Arnold Foundation (2017-22), American Association of Colleges of Pharmacy (2015), Patient-Centered Outcomes Research Institute (2014-16), Cochrane Methods Innovations Fund (2016-18), and UK National Institute for Health Research (2011-14); was an unpaid IMEDS steering committee member at the Reagan-Udall Foundation for the FDA (2016-20), and is an editor at The BMJ. The views and opinions expressed here are those of the author/presenter and do not necessarily reflect official policy or position of the University of Maryland.

Summary

Salary from University of Maryland & The BMJ
Public, foundation, and non-profit funding of academic research
Reimbursement (e.g. lodging, travel) from non-profits
No industry funding

2 of 32

1. Use contributes to herd immunity

Requires testing ability to prevent infection and interrupt viral transmission

2. Reduce disease severity

Requires testing ability to lower risk of hospitalization, intensive care use, and death

Early-middle 2020:

search for an effective covid-19 vaccine

DESIRED ATTRIBUTES OF AN EFFECTIVE COVID-19 VACCINE

… BUT IN REALITY, TRIALS WERE NOT DESIGNED TO STUDY EITHER OF THESE

TO DETERMINE IF A PRODUCT HAS THESE ATTRIBUTES REQUIRES CLINICAL TRIALS

Source: Doshi P. Will covid-19 vaccines save lives? Current trials aren't designed to tell us. BMJ. 2020 Oct 21;371:m4037. https://doi.org/10.1136/bmj.m4037 PMID: 33087398.

3 of 32

Nov 23, 2020

Moderna Chief Medical Officer Tal Zaks warns on #AxiosOnHBO to not "over-interpret" vaccine results: “They do not show that they prevent you from potentially carrying this virus...and infecting others.”

Source: https://twitter.com/axios/status/1331090810666844161

But it did not actually require Moderna’s CMO to acknowledge this. It was discoverable through a close reading of the trial’s design.

4 of 32

1st question for journalists to ask:�What is a study actually studying?

Trials do not vaguely test “how safe and effective” medicines are.
What specific research question (hypothesis) does this study aim to answer?

Investigators define a research question in order to design the trial
Use the “PICO” tool to clarify the question

After you determine the trial’s research question, ask:

Is the trial addressing the question we need answered? (often, it’s not¹)
What questions are not being addressed by this trial? (often, the important questions¹)
Are trial participants being told? (often, they’re not²)

1: Wieseler et al. New drugs: where did we go wrong and what can we do better? BMJ. 2019 Jul 10;366:l4340. https://doi.org/10.1136/bmj.l4340 2: Doshi et al. Informed Consent to Study Purpose in Randomized Clinical Trials of Antibiotics, 1991 Through 2011. JAMA Intern Med. 2017 Oct 1;177(10):1452-1459. https://doi.org/10.1001/jamainternmed.2017.3820.

5 of 32

Patients/Population What kinds of people are being studied?	Are these patients the ones who we think need the drug?
Intervention & Comparison What is being studied and what is it being compared with? And how is the comparison constructed?	Do the intervention and comparator comprise a pair of realistic choices people choose between? Is this a randomized trial? Is it double-blind?
Outcome(s) being studied (in particular, the “primary endpoint”) How is efficacy being defined?	Is this endpoint what we really want to know about?

Use “PICO” to clarify the research question

6 of 32

EXAMPLE EXERCISE #1�Can Vaccine X stop the spread?

PICO	What you might expect	Reality (covid vaccine trials)
Population	Children and adults	Adults (later, children)
Intervention	vaccine	vaccine
Comparison	Inert placebo, randomized trial	Saline, randomized trial
Outcome(s) of interest (endpoints)	Rate of virus transmission to contacts (e.g. family members)	Lab-confirmed covid (of essentially any severity) No testing of contacts was performed.

Conclusion: there’s a mismatch between actual trial design and needed trial design

7 of 32

EXAMPLE EXERCISE #2�Can Vaccine X save lives?

PICO	What you might expect	Reality (covid vaccine trials)
Population	Highest risk (e.g. frail elderly)	Adults … but no recruiting from nursing homes, no immunocompromised
Intervention	vaccine	vaccine
Comparison	Inert placebo, randomized trial	Saline, randomized trial
Outcome(s) of interest (endpoints)	Death rate	Lab-confirmed covid (of essentially any severity)

Conclusion: there’s a mismatch between actual trial design and needed trial design

8 of 32

Where can one determine study design?

Where	When is it available	Notes
Study protocol (ideal)	Written before trial begins	Often not publicly accessible until later (if ever)
Trial registers (e.g. ClinicalTrials.gov, ICTRP, others)	Record typically established before trial begins	May lack sufficient detail to fully clarify PICO
FDA Advisory Committee meetings & related “dockets”	48 hours before advisory committee meets	Search meeting transcripts + investor and FDA slides
Sponsor press releases and investor briefings	Before, during, after trial	See guide¹
Financial documents (in particular SEC filings in the United States)	Filed quarterly (Form 10-Q) and Annually (Form 8K)	See guide¹

1: Tanveer S. “Trial transparency in real time” https://restoringtrials.org/2020/10/18/trial-transparency-in-real-time/

9 of 32

Sample study protocol

Most comprehensive source of trial design details – but often treated by companies as commercially confidential

TRIAL PROTOCOLS – TYPICALLY CONTAIN THE MOST DETAIL

Source: Pfizer. PDF p.158 https://www.nejm.org/doi/suppl/10.1056/NEJMoa2027906/suppl_file/nejmoa2027906_protocol.pdf

10 of 32

A lack of transparency is hard to defend

New York Times, Sept 13, 2020

New York Times, Sept 17, 2020

New York Times, Sept 19, 2020

Moderna

Pfizer

AstraZeneca US trial

Links at https://restoringtrials.org/covid-19/

11 of 32

Source: https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-december-10-2020-meeting-announcement

FDA ADVISORY COMMITTEE MEETINGS

12 of 32

Advisory Committee Meetings

EVENT MATERIALS INCLUDE

Meeting agendas
FDA presentations
Sponsor presentations
Meeting transcripts
and more…

FDA ADVISORY COMMITTEE MEETINGS

13 of 32

FDA ADVISORY COMMITTEE MEETINGS

Source: FDA. https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-october-22-2020-meeting-announcement

click

14 of 32

Source: FDA. https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-october-22-2020-meeting-announcement

FDA ADVISORY COMMITTEE MEETINGS

15 of 32

“FDA is establishing a docket for public comment on this meeting. The docket number is FDA-2020-N-1898…."

Sources: https://www.fda.gov/advisory-committees/advisory-committee-calendar/vaccines-and-related-biological-products-advisory-committee-december-10-2020-meeting-announcement and www.regulations.gov

“DOCKETS” for FDA ADVISORY COMMITTEE MEETINGS

16 of 32

Comment from J. Patrick Whelan MD PhD (University of California, Los Angeles)

Dec 9, 2020�

“I am concerned about the possibility that the new vaccines … have the potential to cause microvascular injury to the brain, heart, liver, and kidneys in a way that does not currently appear to be assessed in safety trials of these potential drugs.”

Source: https://www.regulations.gov/document/FDA-2020-N-1898-0246

EXAMPLE COMMENT FROM FDA ADVISORY COMMITTEE “DOCKET”

17 of 32

Source: From “Trial transparency in real time” how-to guide at https://restoringtrials.org/2020/10/18/trial-transparency-in-real-time/ showing a slide shown on a Moderna investors’ call.

INVESTOR BRIEFINGS

18 of 32

Source: From “Trial transparency in real time” how-to guide at https://restoringtrials.org/2020/10/18/trial-transparency-in-real-time/ showing a slide shown on a Moderna investors’ call.

FINANCIAL FILINGS TO THE SEC (SECURITIES AND EXCHANGE COMMISSION)

19 of 32

Locating trial data/documents, including results

Regulators

Drugs@FDA (https://www.fda.gov/drugsatfda)
FDA Advisory Committees (https://www.fda.gov/advisory-committees)
Health Canada portal (https://clinical-information.canada.ca)
EMA Clinical Data Publication portal (https://clinicaldata.ema.europa.eu)
Japan PMDA (English & 日本語) (https://www.pmda.go.jp/)

Supplemental files for trial publications (for protocol and statistical analysis plan)

See “How to access clinical trial data” guide at https://restoringtrials.org/accessing-trial-data/

AFTER REGULATORY AUTHORIZATION/LICENSURE

20 of 32

Thank you��Contact�pdoshi@bmj.com�pdoshi@rx.umaryland.edu

21 of 32

Source: FDA. https://www.fda.gov/patients/learn-about-drug-and-device-approvals/drug-development-process

22 of 32

“Before filing a marketing application, a developer must have adequate data from two large, controlled clinical trials.”

“Length of Study: 1 to 4 years”

Source: FDA. https://www.fda.gov/patients/drug-development-process/step-3-clinical-research

23 of 32

Reality (non-orphan drugs)

Short. 7% had at least 1 trial ≥12 months; 67% approved with no pivotal trial ≥6 months

33% of indications approved on single pivotal trial (median 2; IQR 1 to 3)

Rhetoric

Long (1 – 4 years)

2 pivotal trials

188 novel agents

16% orphan

84% non-orphan

Downing et al. JAMA. 2014;311(4):368-377.

doi:10.1001/jama.2013.282034

What do we know about a drug at time of approval?

Answer: far less than many assume

24 of 32

“A single trial justified approval for 19 of 46 (41%) of the approved products.”

https://academic.oup.com/healthaffairsscholar/article/1/2/qxad028/7226406

25 of 32

https://www.bmj.com/content/381/bmj.p1048

26 of 32

https://reports.statnews.com/products/failed-trial-successful-drug?variant=40178917245031

27 of 32

Source: Finn A, Malley R. “A Vaccine That Stops Covid-19 Won’t Be Enough” https://nyti.ms/2Fwr098

24 August 2020

“One cannot assume that a vaccine that prevents the development of Covid-19 in a patient will necessarily also limit the risk that the patient will transmit SARS-CoV-2 to other people.”

28 of 32

“None of the trials currently under way are designed to detect a reduction in any serious outcome such as hospital admissions, use of intensive care, or deaths. Nor are the vaccines being studied to determine whether they can interrupt transmission of the virus.”

“Our trial will not demonstrate prevention of transmission,” [Moderna CMO] Zaks said, “because in order to do that you have to swab people twice a week for very long periods, and that becomes operationally untenable.”

21 October 2020

Source: Doshi P. Will covid-19 vaccines save lives? Current trials aren't designed to tell us. BMJ. 2020 Oct 21;371:m4037. https://doi.org/10.1136/bmj.m4037 PMID: 33087398.

This was discoverable through a close reading of each trial’s design

29 of 32

https://restoringtrials.org/regulatory-resources/

30 of 32

CSRs in the public domain

EMA (through Policy 0070)

Health Canada (Vanessa’s Law)

Japan PMDA (clinical and�non-clinical summaries)

https://clinicaldata.ema.europa.eu/

31 of 32

CSRs in the public domain

EMA (through Policy 0070)

Health Canada (Vanessa’s Law)

Japan PMDA (clinical and�non-clinical summaries)

https://clinical-information.canada.ca/

32 of 32

CSRs in the public domain

EMA (through Policy 0070)

Health Canada (Vanessa’s Law)

Japan PMDA (clinical and�non-clinical summaries)

https://www.pmda.go.jp/PmdaSearch/iyakuSearch/

申請資料