Test Taking Skills for Advisors

Faculty Meeting Jan 10, 2019

Presented by Heather Paladine and Anita Softness

Outline for this session

1. Understand your resident’s test taking history and skills
2. ITE Background
3. How to approach the ITE and advise your resident
4. Interpreting and applying the Score Report
5. HOW TO DEVELOP A STUDY PLAN
6. Assessing learning styles and identifying resources
7. ABFM and other educational resources

​

​

Get to know your advisee!

• Learn about your advisee’s history of test taking
• Resources to interpret USMLE and COMLEX scores
• USMLE score interpretation:  https://www.usmle.org/pdfs/transcripts/USMLE_Step_Examination_Score_Interpretation_Guidelines.pdf
• For step 1, mean is 229 and SD 20, minimum passing score ~194
• For step 2, mean is 242 and SD 17, minimum passing score ~209
• How to convert COMLEX scores to percentiles: http://www.nbome.org/CBT_Score_Conv/
• For level 1, mean is 520 and SD 85
• For level 2, mean is 540 and SD 85
• Minimum passing score is set at 400
• Anticipate how much support your advisee may need before ITE

USMLE example

• Step 1 score is 220, Step 2 score is 265
• Step 1: 30^th percentile (exactly at the mean)
• Step 2: 93^rd percentile (1 SD above the mean)
• ABFM score predictor: 98% chance of passing the FM boards based on step 2 score
• Good to explore the resident’s test-taking experience, but this resident likely should continue to use the study skills that made them successful as a student

​

COMLEX example

• Level 1 score is 460, level 2 score is 440
• Level 1: 26^th percentile, less than 1 SD below the mean
• Level 2: 28^th percentile, more than 1 SD below the mean
• ABFM score predictor: 91% chance of passing the FM boards based on level 2 score
• This is a resident who will probably need a study plan in order to do well on standardized tests.
• The resident also scored lower on Level 2, which is more representative of step 3 and the FM boards

​

Info about In-training Exam

• Written by ABFM : “the purpose … is to provide an assessment of each resident's progress, while also providing programs with comparative data about the program...”
• 240 multiple-choice questions, content outline is identical to the ABFM Certification Exam (see next slide)
• Scored using statistical analyses similar to the ABFM Certification Exam. However, no passing score, since the ITE is given to assess progress.
• Residents show gains each year; biggest gain is between yrs 1 - 2.

Content Outline

Cardiovascular 12%

Endocrine 8%

Gastrointestinal 7%

Hematologic/Immune 3%

Integumentary 6%

Musculoskeletal 12%

Nephrologic 3%

Neurologic 3%

​

Nonspecific 9%

Psychogenic 7%

Reproductive: Female 4%,Male 1%

Respiratory 13%

Special Sensory 2%

Population-based Care 5% (biostats and epi, evidence-based med, prevention, health policy and legal issues, bioterror, QI, and geographic/urban/rural issues.)

Patient-based Systems 5% (clinical decision-making, communication and doctor-patient interaction, family and cultural issues, ethics, palliative care, and end-of-life care.)

​

How to approach the ITE

• an opportunity to collect useful data: formative not summative
• use the scoring as guidance to assess a resident’s ability to pass Boards and to determine what resources the resident needs
• reassure resident that it is a low stakes test,-- however needs to be taken seriously – get there on time, answer all the questions, etc.

​

• The resident’s score is at the top right
• The national mean for their class is in the top center.
• The bars represent how well they did in each area. Bars to the left of the red line represent weaker areas, so these should be a focus of study.
• The MPS at the bottom is the minimum passing score for the Family Medicine board exam. A bar to the right represents a resident who would be definitely expected to pass the exam if they took the exam right now (as opposed to the Bayesian score predictor, which represents the resident’s chance of passing the board exam at the end of their third year.)

Predicting Board Performance

• Goal is >= 95% chance of passing the Family Medicine boards, this number is extrapolated from their ITE performance over all years of residency (and USMLE/COMLEX scores step 2 scores) to where they should be as a third year
• This information comes from the ABFM’s Bayesian Score Predictor:
• https://cbas.theabfm.org/BayesianPrediction.aspx

�

This is what the score predictor looks like before entering a resident’s scores.

This is what the score predictor looks like after enter a resident’s scores for step 2 and the ITE scores for first and second year of residency.

Score Interpretation

• Green Zone ≥ 95%: keep doing what you are doing
• Yellow 90-94%: advisor develops study plan with resident
• Red below 90%: CCC and PD develops Study plan

with resident and involves advisor to implement

Addressing Extenuating Circumstances

If the resident feels their score doesn’t reflect their true ability (night float, sick, etc) or if they have difficulty with time, we can arrange for them to take a previous year’s ITE under actual exam conditions (timed with a proctor)

�If a resident has persistent issues with time management or other test-taking problems, they should consider an evaluation for a learning issue. The ABFM may give them more time or other accommodations on the actual board exam.

�

How to develop a study plan

​

• Resident should have their score report and be able to identify subject areas that they are weaker in.
• Ask the resident to choose 2-3 subject areas they are weaker in, and focus their plan around those subjects. (Clinical categories may be more relevant than organ systems.)�

Study Plan

• Look at the ABFMs videos which describe the time and elements of a good study plan to improve test scores:
• https://www.theabfm.org/moc/exampreparation.aspx

��

Timeline and Content of Study Plan

• Taking test questions by itself does not improve scores. However, test questions can be part of a comprehensive study plan IF the critiques are reviewed in depth.
• The resident needs an ongoing education plan such as reading articles or listening to podcasts.
• The ABFM doesn’t recommend cramming, and in general board review courses are only valuable far enough ahead of time to give the learner a long-term study plan. (This also means that cramming in the week before the test is not helpful.)

​

Follow up is key

• Need to create a clear time line and calendar with agreed upon check in dates
• Then don’t forget to check in!

Consider Learning Style

Have the resident do a self-assessment of learning style:

http://vark-learn.com/the-vark-questionnaire/

• V: Visual
• A: Auditory
• R: Reading/Writing
• K: Kinesthetic

​

Most people have mixed learning styles

��

​

Examples of Study Approach for the Visual Learner

• Visual: Make color coded notes, draw diagrams
• Take notes during lectures
• Highlight information while reading
• Avoid auditory distractions when studying

​

Examples of Resources for the Auditory Learner

​

• Podcasts (see list later in presentation)
• Form a resident study group and quiz each other
• Can make their own recording of notes and listen to it�

Examples of Resources for the Reader/Writer

• Read review articles from AFP, Dynamed, UTD
• ABFM KSA modules (residents who take more modules do better on the board exam, and they can choose modules that correspond with their areas of weakness)
• https://portfolio.theabfm.org/Login.aspx?ReturnUrl=%2fMCFP%2fModules.aspx%3ftab%3dsam&tab=sam

​

�

Examples of Study Approach for the Kinesthetic Learner

• Take notes on what they read or listen to, draw diagrams, color code
• Take frequent, brief study breaks
• Study while exercising
• Make flash cards

�

Resources for ITE and Board Study

​

• AAFP family medicine practice boards questions -- free for members: https://www.aafp.org/cme/cme-topic/all/bd-review-questions.html
• USMLE world family medicine board prep --interactive and it has an option to save the explanations and make flash cards
• The ABFM also has an app with practice questions: https://itunes.apple.com/us/app/exam-prep/id401650093?mt=8 (key is to remind them to study the questions they get wrong and not just take questions like popcorn)
• ABFM modules now called KSAs https://portfolio.theabfm.org/Login.aspx?ReturnUrl=%2fMCFP%2fModules.aspx%3ftab%3dsam&tab=sam
• Board Review courses ($$)
• PGY2/3 residents have $500 each (chiefs have $1000) to use for CME, although they can also use it to pay for the board exam

​

Lifelong Learning

• 2 Minute Medicine (free) and NEJM Journal Watch ($) for quick summaries of the latest studies. They provide a few line summary with a longer version if you click the links. 2 Min Medicine sends daily emails. Journal Watch sends weekly updates but you can subscribe to multiple themes (GYN, primary care, hospital medicine) so you may get more frequent emails.
• Medical Letter of Drugs and Therapeutics ($): monthly summaries of new medications, presents evidence on comparative efficacy and price with existing agents, also reviews all medications for various conditions on a regular basis. Has a newsletter and website.
• Essential Evidence Plus (the people that do POEMs) which is a daily critical analysis of an article or guideline using standard analysis techniques. These analyses tend to be primary care oriented. It costs around $85 per year for an individual. 
• Family medicine Study Guide ($13.99) non-profit portable curriculum that includes 75 original case studies, 363 short answer questions, 150 rapid fire questions, 50 "Quick Pix" dermatology cases with 152 related questions and 2hrs 26mins 56 secs of Mini-Podcasts to help motivate, guide and assist you in your studying.

​

PODCASTS

• AAFP podcast (free) https://www.aafp.org/journals/afp/explore/podcast.html
• Primary Care RAP by Hippo Entertainment https://www.hippoed.com/pc/rap/ is entertaining, relevant and interesting, 3 hour monthly podcasts, CME offered, ($395/yr or $95/yr for residents, one hr per month is free)
• Green Journal podcast for updates
• Louisville Lectures (Hospitalist and internal medicine)

​

• Greyscale for medical storytelling
• Curbsiders Internal Medicine and Review of Systems both were highly recommended by int med residents, they are free, but I have not listened to them much

​

​

POC Resources

• Dynamed for workup or treatment. I find it’s more evidence based than Uptodate
• Up to Date tracks my time reading various articles for CME credits.
• iNYP dashboard has POC decision making tools for healthcare maintenance, and links to the statin decision aid which calculates ASCVD risk and translates it into a patient friendly version to explain current risk and reduction with statin
• Memorial Sloan Kettering’s About Herbs website for looking up herbs and herb/disease or herb/drug interactions
• Lactmed for looking up meds in breastfeeding https://toxnet.nlm.nih.gov/newtoxnet/lactmed.htm
• PEPID for smart phones: focuses on primary care problems, and cites primary care evidence.  Has useful calculators, pill identifiers, and a lab test reference tool.  I find it to be more concise than Up to Date.  ($175 per year)

​

POC Apps

• OB Dating wheel
• Aspirin guide app
• Urgent OB
• AHRQ EPS
• CDC contraception
• Visual dx 
• Diagnausaurs

​

• Shots
• ASCCP ($10)
• MDCalc 
• Epocrates 
• About herbs 
• eGFR 
• Medscape 

​