1 of 33

ALACRITY Training Series

Adaptive Measurement of Suicidality & Risk

March 8, 2022

Robert D. Gibbons, Ph.D.�Blum-Riese Professor�The University of Chicago

1

2 of 33

Disclosure

At the request of the NIMH, I founded Adaptive Testing Technologies which is the company that licenses the CAT-MH™ and the K-CAT™ technology. The terms of this arrangement have been reviewed and approved by the University of Chicago in accordance with its conflict-of-interest policies.

​

www.adaptivetestingtechnologies.com

​

2

3 of 33

What is Measurement?

Measurement is the process of obtaining the magnitude of a quantity relative to an agreed standard.

​

“Not everything that can be counted counts, and not everything that counts can be counted.”

Albert Einstein

​

3

4 of 33

Measurement in the Physical Sciences

Silver in parts per trillion (1 second in 32,000 years)

4

5 of 33

Measurement in the Social Sciences

5

6 of 33

Classical vs. IRT Measurement

Classical Measurement Model

6

7 of 33

Classical vs. IRT Measurement

Item Response Theory (IRT)

7

8 of 33

What is CAT?

Arithmetic Algebra Calculus

​

Imagine a 1000-Item Math Test

8

9 of 33

The Bifactor Model

Preserves multidimensionality

Severity

Uncertainty in severity estimate

Primary

Secondary

Thresholds

Discrimination

Gibbons and Hedeker, 1992, Psychometrika Gibbons et.al., 2007, Applied Psychological Measurement

9

10 of 33

What are the CAT-MH™ & K-CAT®

  • A suite of 15 computerized adaptive tests (CAT) based on multidimensional item response theory (MIRT).
  • This means that we can, for example, extract the information from 400 depression symptom-items using an average of 10 adaptively administered items, yet maintain a correlation of r>=0.95 with the 400-item test score.
  • As such, we can dramatically increase precision while eliminating clinician burden and minimizing subject burden.
  • The K-CAT® extends this technology to youth ages 7-17.

10

11 of 33

Advantages of the CAT-MH™ & K-CAT®

  • Provide a constant precision of measurement throughout the entire severity continuum.
  • Items are targeted to a patient’s specific level of severity at that point in time.
  • Different questions are asked upon repeat administration, eliminating response bias produced by repeatedly asking the same questions (weekly, daily, hourly assessments).
  • Validated against structured clinical diagnostic interviews (e.g., SCID-DSM-5) to provide diagnostic profiles for many disorders and suicide risk.
  • Can be used for both screening (identify homogeneous group of patients for a trial) and measurement (as a longitudinal outcome measured hourly, daily, weekly) during the trial.

11

12 of 33

Advantages of the CAT-MH™ & K-CAT®

  • Ideal for longitudinal assessments essential for measurement-based care.
  • Estimates the severity of a disorder (e.g., depression) as well as estimating the uncertainty in that severity score and measuring all patients to the same level of precision.
  • Cloud-based and scalable to any size population via HIPAA secure AWS platform. Patients can be screened, measured, and monitored in or out of the clinic.
  • Can be fully integrated into the electronic health record (e.g., Epic) for a patient with already developed clinical workflows designed to integrate into behavioral health and primary care practices and clinics.

12

13 of 33

How Do Adaptive Tests Work?

ADMINISTER QUESTION WITH MEDIUM SEVERITY

ESTIMATE SEVERITY BASED ON RESPONSE TO INITIAL QUESTION

SELECT NEXT MOST INFORMATIVE QUESTION

END QUESTIONS WHEN DESIRED PRECISION OF MEASUREMENT IS REACHED

13

14 of 33

What can the CAT-MH™ and K-CAT® Measure?

Adult (English, Spanish, and Chinese)

  • Depression
  • Anxiety
  • Mania/Hypomania
  • Suicidality
  • PTSD
  • Substance Use Disorder
  • Psychosis
  • Adult ADHD
  • Social Determinants of Health

Pediatric ED (English and Spanish)

  • Screen for Suicidal Youth (CASSY)

​

Perinatal, Criminal Justice, ED, LGBTQ

  • Depression, Anxiety, Suicidality, SUD, …

Youth Ages 7 to 17 (English and Spanish)

  • Depression
  • Anxiety
  • Mania/Hypomania
  •  ADHD
  • Conduct disorder
  • Oppositional defiant disorder
  • Suicidality
  • Substance Use Disorder

14

15 of 33

CAT Suicide Scale (CAT-SS)

    • Map depression, anxiety and mania items onto suicidal ideation and behavior.
    • 111 out of 1008 items qualified.
    • 10 CAT items, r=0.96 with the 111 item score (110 sec).
    • Precision level = 5 points on a 100 point scale.
    • Validation UC and UMass EDs – n=300.
      • 52-fold increase in clinician rated suicidal ideation on C-SSRS across range of CAT-SS.
      • No risk vs. high risk (CAT-SS) vs any ideation (C-SSRS) sensitivity=1.00, specificity=0.95, kappa=0.81

Gibbons, Kupfer, Frank Moore, Boudreaux JCP, 2018

15

16 of 33

Computerized Adaptive Tests for Rapid and Accurate Assessment of Psychopathology Dimensions in Youth 

Robert D. Gibbons, PhD, David J. Kupfer, MD, Ellen Frank, PhD, Benjamin B. Lahey, PhD, Brandie A. George-Milford, MA, Candice L. Biernesser, LCSW, MPH, Giovanna Porta, MS, Tara L. Moore, MA, MPH, Jong Bae Kim, PhD, David A. Brent, MD 

Journal of the American Academy of Child & Adolescent Psychiatry 

​

DOI: 10.1016/j.jaac.2019.08.009

Copyright © 2019 American Academy of Child and Adolescent Psychiatry Terms and Conditions

17 of 33

The K-CAT

  • 5-year NIMH Funded Study

​

  • David Brent Clinical PI

​

  • 2400 item bank (1200 parent, 1200 child)

​

  • Depression, anxiety, mania, ADHD, ODD, CD, suicidality

​

  • 7 minutes for parent and child.

​

  • Validated based on clinical interviews (KSADS) of 1000 parent-child pairs.

​

  • Diagnostic prediction system and dimensional severity measurement system

​

​

​

17

18 of 33

  • Depression Domain
    • 177 items
    • Suicide subdomain – 10 items
    • 801 subjects ages 7-18
  • Results
    • K-CAT-SS - AUC=0.95 for structured clinical interview.
    • Full K-CAT™- AUC=0.996
    • Test-retest reliability 0.81
    • Average of 7 items

​

CAT Child Suicide Scale – K-CAT-SS

18

19 of 33

19

20 of 33

Bifactor Model – Examples out of 64 items

Item Threshold Loading Text_______________________________________

28 -0.47 0.65 I wanted to be alone.

14 -0.17 0.80 I felt lonely.

131 0.02 0.84 I felt blah and low down.

59 0.18 0.82 I felt worthless.

132 0.18 0.88 I felt like I was losing it.

2 0.33 0.71 I felt like I didn't have any friends.

115 0.36 0.88 I felt listless compared to how I used to be.

127 0.45 0.93 I stopped enjoying anything.

68 0.45 0.74 I thought my family would be better off without me.

20 0.57 0.82 I didn't enjoy anything at all.

174 0.70 0.86 My life wasn't worth living.

69 0.90 0.88 I thought about killing myself.

70 1.19 0.83 I had a plan for how I would kill myself.

30 1.65 0.75 I made a suicide attempt.

​

​

​

20

21 of 33

​

The CASSY - Computerized Adaptive Screen for Suicidal Youth

21

22 of 33

​

The CASSY - Design

  • Universal screen for suicide risk in medical emergency departments
  • Pediatric Emergency Care Applied Research Network (PECARN)
  • 13 geographically diverse US EDs
  • Calibration Phase – 6,344 adolescents
  • Validation Phase
    • Study 1 – 2075 adolescents with 3-month follow-up
    • Study 2 – 2754 adolescents with 3-month follow-up
  • Suicide attempt rates
    • Study 1 – n=102 (4.9%) within 3 months
    • Study 2 – n=165 (6.0%) within 3 months

​

​

22

23 of 33

​

The CASSY – IRT Approach

  • Bifactor Model fitted to the 92-item bank
  • Sub-domains
    • Suicidal ideation and behavior
    • Psychopathology
    • PTSD
    • Social adjustment
    • Sleep
    • Anger/aggression
    • Substance use
  • Simulated CAT based on the bifactor model parameter estimates
    • Mean of 11 items (5-21)
    • r=0.94 correlation with 72 item bank score (20 items removed)
    • Median administration time 1 minute 24 seconds (75th % 2 min 4 sec)

​

23

24 of 33

​

The CASSY – Predictive Accuracy

  • 3-month suicide attempt prediction
    • Study 1
      • AUC=0.89, 95% CI (0.85, 0.91)
      • At specificity of 0.8, sensitivity = 0.83
      • At specificity of 0.9, sensitivity = 0.61
    • Study 2
      • AUC=0.87, 95% CI (0.85, 0.89)
      • Using Study 1 cut-point, specificity=0.73, sensitivity = 0.82

​

24

25 of 33

3/7/2022

The CASSY – Example Test Session

25

26 of 33

VA and Active Military Applications

  • VA: Brenner et.al. (2022), PLOS One, validation of CAT-SS in military veterans
    • For every 10 pt change in CAT-SS:
      • 50% increase in active ideation with plan and intent in next year.
      • 77% increase in attempts in next year.
    • AUC for Active Ideation with plan and intent in next year:
      • Attempt past year AUC = 0.64, CAT-SS AUC=0.84.
    • AUC for Suicide Attempt in next year:
      • Attempt past year AUC = 0.72, CAT-SS AUC=0.88.

​

  • Active Military: Wyman et.al. (2020), JAMA Network, Wingman-Connect
    • Air Force Cluster-randomized suicide prevention RCT.
    • Significantly lower depression (CAT-DI) and suicidal ideation (CAT-SS).
      • Effect Size = -0.23 (-0.39, -0.09).
    • The CAT-SS is sensitive to change.

​

​

26

27 of 33

EPIC Integration

27

28 of 33

Intensive Longitudinal Measurement

Daily Home Monitoring of a Deep Brain Stimulation Patient for 6 months

Sani et.al. , 2017, Translational Psychiatry

28

29 of 33

Funding: The Mental Health and Substance Use Disorders Prevalence Study (MDPS) is a cooperative agreement between RTI International and the Substance Abuse and Mental Health Services Administration (SAMHSA). SAMHSA provided funding for the project (H79FG000030).

Disclaimer: The views and opinions contained in this presentation do not necessarily reflect those of SAMHSA or the U.S. Department of Health and Human Services and should not be construed as such.

The SAMHSA/RTI Mental Health and

Substance Use Disorders Prevalence Study (MDPS)

  • Nationally representative prevalence study of mental health and substance use disorders
  • Uses the CAT-MH™ and CIDI as two 1st stage screeners to determine who gets a SCID
  • Preliminary results for CAT-MH screening in 1st 1,000 participants:
    • Severe Psychosis – 0.9%
    • Moderate and Severe Depression – 8.4%
    • Moderate and Severe Anxiety – 15.6%
    • Severe Mania/Hypomania – 3.2%
    • Intermediate and high risk for Substance Use Disorder - 15.7%
    • PTSD Highly likely – 3.8%

​

  • Remote Administration time = 8.5 minutes

​

29

30 of 33

UCLA STAND Program: Metrics since 2017

​

​

​

​

​

Offered screening

85,893 �UCLA students

​

Registered over�7,317�students for symptom screening

Offered treatment �3,545 �students using innovative model

Responded to over 1,265 �real-time risk alerts for suicidality, etc.

!

Trained over

450 �student coaches�

Severe

Normal/Mild

Severe

Mild

Normal

Normal/Mild

30

31 of 33

Scientific Literature

  1. Gibbons R.D., & Hedeker D.R. Full-information item bi-factor analysis. Psychometrika, 57, 423-436, 1992.
  2. Gibbons R.D., Bock R.D., Hedeker D., Weiss D., Segawa E., Bhaumik D.K., Kupfer D., Frank E., Grochocinski V., Stover A. Full-Information Item Bi-Factor Analysis of Graded Response Data. Applied Psychological Measurement, 31, 4-19, 2007.
  3. Gibbons R.D., Weiss D.J., Kupfer D.J., Frank E., Fagiolini A., Grochocinski V.J., Bhaumik D.K., Stover A. Bock R.D., Immekus J.C. Using computerized adaptive testing to reduce the burden of mental health assessment. Psychiatric Services, 59, 361-368, 2008.
  4. Gibbons R.D., Weiss D.J., Pilkonis P.A., Frank E., Moore T., Kim J.B., Kupfer D.K. The CAT-DI: A computerized adaptive test for depression. Archives of General Psychiatry, 69, 1104-1112, 2012.
  5. Gibbons R.D., Hooker G., Finkelman M.D., Weiss D.J., Pilkonis P.A., Frank E., Moore T., Kupfer D.J. The CAD-MDD: A computerized adaptive diagnostic screening tool for depression. Journal of Clinical Psychiatry, 74, 669-674, 2013.
  6. Gibbons R.D., Weiss D.J., Pilkonis, P.A., Frank E., Moore T., Kim J.B., Kupfer D.J. Development of the CAT-ANX: A computerized adaptive test for anxiety. American Journal of Psychiatry, 171, 187-194, 2014.
  7. Gibbons R.D., Bi-factor Analysis. Encyclopedia of Quality of Life Research, Springer, Netherlands, 386-394, 2014.
  8. Achtyes E.D., Halstead S., Smart L., Moore T., Frank E., Kupfer D., Gibbons R.D. Validation of computerized adaptive testing in an outpatient non-academic setting. Psychiatric Services, 66, 1091-1096, 2015.
  9. Beiser D., Vu, M., Gibbons, R.D. Test-retest reliability of a computerized adaptive depression test. Psychiatric Services, 67, 1039-1041, 2016.
  10. Gibbons R.D., Computerized adaptive diagnosis and testing of mental health disorders. Annual Review of Clinical Psychology, 12, 83-104, 2016.
  11. Kim J.J., Silver R.K., Elue R., Adams M.G., La Porte L.M., Cai L., Kim J.B., Gibbons R.D. The experience of depression, anxiety and mania among perinatal women. Archives of Women’s Mental Health, 19, 94-100, 2017.
  12. Gibbons R.D., Kupfer D., Frank E. Moore T., Beiser D., Boudreaux E. Development of a computerized adaptive suicide scale., Journal of Clinical Psychiatry,78, 1376-1382, 2017.
  13. Gibbons R.D., Beiser D., Boudreaux E., Kupfer DJ. Einstein, measurement and prediction. Journal of Affective Disorders, 256, 674-675, 2016.
  14. Sani S., Busnello J., Kochanski R., Cohen Y., Gibbons R.D. High frequency measurement of depressive severity in a patient treated for severe treatment resistant depression with deep brain stimulation. Translational Psychiatry, 7, e1207, 2017.
  15. Aschebrook-Kilfoy, B., Ferguson, B.A., Angelos, P., Kaplan, E.L., Grogan, R.H., Gibbons, R.D. Development of the ThyCAT: A clinically useful computerized adaptive test to assess quality of life in thyroid cancer survivors. Surgery, 163, 137-142, 2017.
  16. Gibbons. R.D., Alegria, M., Cai, L., Herrera, L., Markle, S.L., Collazo, F. Garcia, E.B. Successful validation of the CAT-MH scales in a sample of Latin American migrants in the U.S. and Spain. Psychological Assessments, 30(10), 1267-1276, 2018.
  17. Graham A.K., Minc A., Staab E., Beiser D.G., Gibbons R.D., Laiteerapong N. Validation of a computerized adaptive test for mental health in primary care. Annals of Family Medicine, 17, 23-20, 2019.
  18. Beiser, D.J. Ward, C.E., Vu, M. Laiteerapong, N., Gibbons, R.D. Depression in Emergency Department Patients and Association with Healthcare Utilization. Academic Emergency Medicine, 26(8), 878-888, 2019.
  19. Gibbons R.D., Smith J.D., Brown C.H., Sajdak M., Tapia N., Kulik A., Epperson M.W., Csernansky J. Improving the evaluation of adult mental health disorders in the criminal justice system using computerized adaptive testing. Psychiatric Services, 70(11), 1040-1043, 2019.
  20. Gibbons R.D., deGruy F.V. Without wasting a word: Extreme improvements in efficiency and accuracy using computerized adaptive testing for mental health disorders (CAT-MH). Current Psychiatry Reports. 21, 1053-9, 2019.
  21. Gibbons R.D., Beiser D.G., Boudreaux E.D., Kupfer D.J. Einstein, measurement, and prediction. Journal of Affective Disorders, 256, 674-75, 2019.
  22. LaPorte L.M., Lim J.J., Adams M.G., Zagorsky B.M., Gibbons R.D., Silver R.K. Feasibility of perinatal mood screening and text messaging on patients’ personal smartphones. Archives of Women’s Health. 23, 181-188, 2020.
  23. Gibbons R.D., Kupfer D., Frank E., Lahey B.B., George-Milford B., Biernesser C., Porta G., Moore T., Kim J.B., Brent D. Computerized adaptive tests for rapid and accurate assessment of psychopathology dimensions in youth. Journal of the American Academy of Child and Adolescent Psychiatry, 59(11), 1264-1273, 2020.
  24. Guinart D., de Filippis R., Rosson S., Patil B., Prizgint L., Talasazan N., Meltzer H., Kane J.M. and Gibbons R.D. Development and Validation of a Computerized Adaptive Assessment Tool for Measurement of Psychotic Symptoms. Schizophrenia Bulletin, sbaa168, 2020.
  25. Gibbons R.D., Alegria M., Markle S., Fuentes L., Zhang L., Carmona R., Collazos F., Wang Y., Baca-García E. Development of a computerized adaptive substance use disorder scale for screening and measurement: the CAT-SUD. Addiction, 115(7):1382-1394, 2020.
  26. Wyman P.A., Pisani A.R., Brown C.H., Yates B., Morgan-DeVelder L., Schmeelk-Cone K., Gibbons R.D., … Effect of Wingman-Connect Upstream Suicide Prevention for Air Force Personnel in Training: A Cluster Randomized Controlled Trial. JAMA Network Open, 3(10):2020.
  27. King C.A., Brent D., Grupp-Phelan J., Casper T.C., Dean J.M., Chernick L.S., Fein J.A., … Gibbons R.D. Computerized Adaptive Screen for Suicidal Youth (CASSY): Prospective Development and Validation. JAMA Psychiatry. Published online ahead of print, 2021.
  28. Mustanski B., Whitton S., Newcomb M., Clifford A., Ryan D.T., Gibbons R.D. Predicting suicidality using a computer adaptive test: Two longitudinal studies of sexual and gender minority youth. Journal of Consulting and Clinical Psychology, 89(3), 166-175, 2021.
  29. Berona J., Whitton S., Newscomb M.E., Mustanski B., Gibbons R.D. Prospective risk and protective factors for the transition from suicide ideation to attempt among sexual and gender minority youth. Psychiatric Services. In Press, 2021.
  30. Gibbons R.D., Kupfer D.J., Frank E., Brent D.A., Dr. Gibbons et al. Reply, Journal of the American Academy of Child & Adolescent Psychiatry. 60(5), 542-543, 2021.
  31. Gibbons R.D., Chattopadhyay I., Meltzer H., Kane J.M., and Guinart D. Development of a computerized adaptive diagnostic screening tool for psychosis. Schizophrenia Research. Published online ahead of print, 2021.
  32. Karpin J.E., Rodriguez T.G., Traboulsi C., Rai V., Gibbons R.D., and Rubin D.T. Assessment of Comorbid Depression and Anxiety in Inflammatory Bowel Disease Using Adaptive Testing Technology. Observations and Research, 3(1), 1-7, 2021.
  33. Grunebaum M.F., Mann J.J., Galfalvy H.C., and Gibbons R.D. Computerized-Adaptive vs. Traditional Ratings of Depression and Suicidal Thoughts: An Assay Sensitivity Pilot Study in a Ketamine Clinical Trial. Frontiers in Psychiatry. 12, 450, 2021.
  34. Wenzel E.S., Gibbons R.D., O’Hara M.W., Duffecy J., Maki P.M. Depression and Anxiety Symptoms Across Pregnancy and the Postpartum in Low-Income Black and Latina Women. Archives of Women’s Mental Health. Published online ahead of print, 2021.
  35. Brenner L.A., Betthauser L.M., Penzenik M… Gibbons R.D. et al. Development and Validation of Computerized Adaptive Assessment Tools for the Measurement of Posttraumatic Stress Disorder among United States Military Veterans. JAMA Network, 4(7) 2021.

​

31

32 of 33

32

33 of 33

We spend billions on biological measurements, yet we validate them using stone-age clinical measurements.

​

​

“Statistics… is the most important science in the whole world, for upon it depends the practical application of every other (science) and of every art.”

33