Using pilot project to ground your NIH proposal

• Jude P. Mikal, PhD
• PCHS / C2DREAM
• UMN College of Pharmacy
• ID Core – C2DREAM

What this talk will cover

Pilot data

• Pilot data are preliminary results generated from small-scale studies (or as part of a larger-scale study) designed to assess feasibility, refine methods, estimate parameters, and inform the development of a larger, fully powered research project.
• The purpose of a pilot data section is to:
• Reduce a reviewer’s perception of risk
• Demonstrate the potential of a researcher and of a project
• Show methodological feasibility or proof-of-concept

​

From pilot to proposal: how much pilot data is enough?

• There are different mechanisms within and between agencies - each with its own expectations for pilot data
• Logistics and pragmatics play a significant role:
• NIH standard deadlines
• PAR / PAS / RFA
• Researchers rarely move from pilot -> results -> NIH proposal. They may:
• …submit an NIH proposal, get rejected and then extract a small pilot project to increase competitiveness
• …submit an NIH before a pilot study is completed, get rejected, complete pilot and resubmit with more robust pilot data
• With the housekeeping out of the way, we can turn to the real reason you’re here: using pilot data to strengthen and de-risk your NIH proposal.

Using pilot data to de-risk your proposal

• In order to de-risk your project, first ask yourself what parts of your project are most likely to raise eyebrows among reviewers
• Use your pilot data to buttress those elements of your proposal that seem most risky. These features can be found across multiple domains:
• Conceptual readiness
• If you are using novel concepts, theories, or interventions: how does your pilot data demonstrate clear proof-of-concept and potential for success?
• Methodological readiness
• If you are using novel data, measures or methods: can your pilot data support the use of those tools in other, related contexts?
• Anticipatable logistical challenges
• If you are working with new populations or new teams: how can you demonstrate a clear track record of success with those individuals?
• Funding readiness
• As you transition from smaller to larger pots of money, agency and reviewer willingness to assume risk is generally reduced. Does your pilot data support the resources you are requesting with commensurate promise of forward progress in both health and science?
• NB: Not all projects will have pilot data for all four elements and that’s OK! Not all projects will have risk across all four elements!

Procedurally, what does this look like?

• Compile your list of concepts, methods, data and other features that may appear novel or risky. Note that there may be multiple features in a single domain (e.g., two novel methodological approaches)
• Index the pilot and preliminary data that you have to demonstrate that you have successfully studied, deployed, or published on each feature-of-interest in the past
• Pilot and preliminary data need not necessarily only include published studies. You can also use:
• Unpublished or preliminary results
• Other project deliverables like reports and datasets
• Qualitative interviews and interview quotes
• Population pools, community partnerships … be creative!
• If it speaks to the feasibility and potential success of your project, shop it around to colleagues! Just because we haven’t used it yet – doesn’t mean it is not useful!

Pro tip – be sure to preen!

• Be judicious in selecting features that require proof-of-concept – only features that are central to your project deserve “air time”
• Studies can come from any member of the research team
• Studies do not have to be topically relevant – if they are methodologically relevant, and vice versa
• Be judicious in selecting studies or other project deliverables to de-risk your project
• An exercise: select one feature from your project that may appear risky to your reviewers and do a brain dump. What published work might help to de-risk this for reviewers? What unpublished work might you have? What datasets? What quotes from qualitative interviewers? And then what is one potentially ridiculous-seeming thing that might lend some credibility to this element of the project?
• Now pair and share!

Some decisions to make before you begin:

• Does it make more sense to include your pilot data as a single section – or beneath each aim?
• Are there exciting results that demonstrate proof-of-concept for the whole project? Or is your preliminary data section a smorgasbord of different studies that function together to support individual aims, methods, or concepts?
• Does it make most sense to move study to study – or concept / feature to concept / feature?
• How much space will you allocate to (1) your pilot / preliminary data, and (2) to each study or concept you present? Remember, “air time” should be a function of importance.
• Protip: resist the temptation to simply present success stories. Focus on communicating proof-of-concept, methodological feasibility, opportunities for growth or improvement of a particular study, promising new directions or insights that are unique to your team because you’ve completed these preliminary studies!

​

Pilot data: a study-by-study approach

For your research study, be sure to include the following:

• A short title or subheading
• A brief research objective describing study goals
• Methodological information about the study. Focus specifically on:
• Data or participants
• Core procedures and methods
• Results
• Deviations or lessons learned
• Avoid the temptation to be exhaustive, include elements that establish feasibility or “de-risk” your proposal

​

​

​

For example

• Project X: a case for data collection feasibility
• Project description: A groundbreaking initiative through the University of Minnesota, Project X aims to …
• Methods: Researchers recruited 105 individuals from diverse communities throughout Minnesota to study the impacts of x exposures on later life y outcomes. Samples were taken from … and analyzed using …
• Results: Preliminary results showed a robust relationship between … and … providing insights into ... and laying the foundation for …
• Relevance to the Current Proposal: Our successful deployment of x methods helped us to establish community connections for … and through those connections our team reached a 50% response rate with 90% of participants remaining active in the study for the full 6-month intervention…

Pilot data: a study-by-study approach

Once you have adequately summarized the nuts-and-bolts of your research, consider subheadings that draw attention to …

• …methodological feasibility
• Specifically: recruitment, retention, adherence to treatment / procedures, task completion, timelines, data quality
• Example: Our 90% retention rate between waves 1 and 2, shows robust follow-up capacity and evidence of participant engagement.
• …promise or proof-of-concept
• Specifically: preliminary associations and effect sizes, validated measures, or model output
• Example: Preliminary analyses showed that participants who completed [intervention] reported consistent reductions in perceived stress, suggesting the expected directionality and informing power estimates for Aim 1
• …opportunities for growth
• Specifically: what was it about your previous work that keeps it from being the definitive answer to your RQ?
• Pro Tip: Where possible, link de-risking evidence to specific concepts, methods or aims!

Pilot data: a study-by-study approach

Might you want to include …

• An introduction?
• Can provide a broad, lay-of-the-land
• Can provide a road-map for the preliminary / pilot data section including important concepts or methods
• A conclusion?
• Can provide a broad-level synthesis of remaining gaps, lingering questions, and directions for future research
• Are they worth the space?

All at once, or peppered throughout?

• Space allocation:
• NIH Recommendation: 25% of your proposal, 3pp for an R01, 1.25 pages for an R03 / R21
• Observation IRL: 10%, 1.25 pages for an R01, 0.50 pages for an R03 / R21
• Revisit your decisions from slide 8 and ultimately use enough string to tie the package
• That said, do not be shy in demonstrating your project’s likelihood of success. Pilot data can be mentioned in your
• ...aims: usually a cursory mention of “promising pilot data from our own investigative team”
• …significance: more detail, reviews important concepts or theories
• …innovation: not often mentioned here, but if there are groundbreaking new approaches being applied in novel ways – this might be appropriate
• …approach: novel methods, data, populations or approaches or novel applications of any of the above
• It is OK to mention your pilot data multiple times, but do not repeat text. Use each section to highlight something different!

Pitfalls: when you might consider waiting

• Preliminary data contradicts your main research hypothesis – and you cannot explain why
• The pilot data does not sufficiently de-risk your project
• Your methods or measures lack evidence or validation
• Your conceptual or model is shifting
• You are proposing to expand an intervention with limited demonstrable effectiveness
• Your pilot data exposed a key weakness or limitation you are unable to address

​

​

What are some other challenges you might face in either collecting pilot data, assembling a portfolio of pilot research, or presenting your pilot research?

​

​

Questions and Comments