1 of 22

Evaluating NCDOT’s Research Selection Process

By: Curtis T. Bradley, Ph.D.

January 10, 2022

2 of 22

Table of Contents

  • Background
    • Problem Statement
    • Research Program Development Process
  • Methodology and Data Analysis
  • Conclusions, Improvements and Next Steps
  • Questions

TRB Presentation

2

3 of 22

Background

TRB Presentation

3

4 of 22

Problem Statement

TRB Presentation

4

Research Selection: Two Phases

    • Research Ideas
    • Proposals

Evaluate Selection Methodology

    • Scoring
    • Selection
    • Consistency

5 of 22

# of Research Ideas / Proposals / Awards

TRB Presentation

5

6 of 22

6

Research Program Annual Timeline

7 of 22

Project Selection Oversight

7

Research and Development Unit Oversees Overall Program

Solicits Ideas

Research Engineers Manage Projects

Coordinate all Activities

Research Subcommittees Review and Recommend Proposals for Funding

Environmental

Structures and Geotech

Pavement, Materials, Maintenance

Planning, Policy and Transit

Traffic, Mobility and Safety

Research Executive Committee Approves Work Program

Senior Management

Executives

8 of 22

Research Idea Criteria

TRB Presentation

8

Need / Urgency / Importance

Probability of Implementation

Tangible Benefits

9 of 22

Nine Proposal Criteria

2020 and Earlier

  • Understanding of Problem Statement
  • Organization of Proposal
  • Quality of Literature Review
  • Research Methodology
  • Proposed Data Collection Techniques
  • Proposed Analysis Methodology
  • General Readability
  • Grammar, Spelling and Punctuation
  • Proposal Cost vs. Value

​

TRB Presentation

9

2021 and Later

  • Understanding of Problem Statement
  • Need / Urgency
  • Quality of Literature Review
  • Research Methodology
  • Project Implementation Plan
  • Creative / Innovative
  • General Readability
  • Experience / Expertise
  • Proposal Cost vs. Value

​

10 of 22

Methodology and Analysis

TRB Presentation

10

11 of 22

Literature Review - Lessons Learned

TRB Presentation

11

Regression Modeling (Coefficients)

Subject Matter Expert Qualifications

Benefit Cost Analysis

Funding Source

Qualitative vs. Quantitative (Metrics)

12 of 22

Statistical Methodology

TRB Presentation

12

t-test

    • Evaluating the differences between the mean scores of selected vs. not selected

Logistic Regression

    • Evaluating the relationship between the selection of ideas / proposals to the established criteria.

Linear Regression

    • Evaluating the relationship between % of support vs. criteria scores

Correlation Analysis

    • Evaluating the relationships between the independent variables

Variance Inflation Factor

    • Identify evidence of multicollinearity

α= 0.05 (Meaning any p-value that is less that 0.05 is considered statistically significant)

13 of 22

RESEARCH IDEA EVALUATION (t-test)

13

​

​

Selected

Not Selected

​

p-value

Mean

Median

Mode

Mean

Median

Mode

Urgency

0.00

3.56

3.50

3.50

2.90

2.91

3.00

Implementation

0.00

3.71

3.67

4.00

2.94

3.00

3.00

Benefits

0.00

3.75

3.67

4.00

3.02

3.00

3.00

Support

0.00

82%

83%

100%

50%

50%

50%

α= 0.05 (Meaning any p-value that is less that 0.05 is considered statistically significant)

14 of 22

PROPOSAL EVALUATION (t-test)

14

​

​

Selected

Not Selected

​

t-test

Mean

Median

Mode

Mean

Median

Mode

Problem Statement

0.00

4.03

4.00

4.00

3.61

3.71

4.00

Literature Review

0.00

3.76

3.75

4.00

3.47

3.50

4.00

Methodology

0.00

3.75

3.75

4.00

3.47

3.50

3.00

Readability

0.00

3.94

4.00

4.00

3.69

3.71

4.00

Cost vs. Value

0.00

3.60

3.67

3.67

3.25

3.29

3.00

Support

0.00

80%

83%

100%

54%

50%

0%

α= 0.05 (Meaning any p-value that is less that 0.05 is considered statistically significant)

15 of 22

Research Idea Regression

15

​

Estimates

Std. Error

p-value

Intercept

-7.1762

1.8401

0.000

Urgency

0.4344

0.4653

0.351

Implementation

1.7460

0.5330

0.001

Benefits

-0.1191

0.5774

0.837

Yes

0.4342

0.1051

0.000

No

-0.6170

0.1737

0.000

 

Coeff.

Std. Error

p-value

Intercept

-0.4176

0.0575

0.0000

Urgency

0.0794

0.0286

0.0059

Implementation

0.0506

0.0317

0.1120

Benefits

0.1939

0.0345

0.0000

Logistic Regression

Linear Regression

α= 0.05 (Meaning any p-value that is less that 0.05 is considered statistically significant)

16 of 22

Correlation / Variance Inflation Factor

TRB Presentation

16

17 of 22

Proposal Regression

17

Logistic Regression

Linear Regression

​

Estimates

Std. Error

p-value

Intercept

-7.0097

2.4135

0.0037

Problem Statement

0.8374

0.5009

0.0946

Literature Review

1.2689

0.5725

0.0267

Methodology

-0.5920

0.4918

0.2287

Readability

-0.0456

0.5558

0.9346

Cost vs. Value

0.3932

0.3868

0.3093

Yes

0.1824

0.0950

0.0548

No

-0.5601

0.1678

0.0008

 

Coeff.

Std. Error

p-value

Intercept

-0.8123

0.164473

0.0000

Problem Statement

0.2388

0.044687

0.0000

Literature Review

0.0036

0.054637

0.9472

Methodology

0.0604

0.046515

0.1955

Readability

-0.0007

0.054274

0.9904

Cost vs. Value

0.0963

0.036759

0.0095

α= 0.05 (Meaning any p-value that is less that 0.05 is considered statistically significant)

18 of 22

Conclusions, Improvements & Next Steps

TRB Presentation

18

19 of 22

Conclusions

TRB Presentation

​

19

Rating Criteria Evaluation

    • Rating criteria is statistically valid
    • Some criteria are more critical than others
    • Higher numeric ratings tend to correlate with selected projects

Research Idea / Proposal Analysis

    • Probability of Implementation / Problem Statement / Literature Review / Cost vs. Value
    • Reviewer Support is very important.

20 of 22

Improvements to Process

TRB Presentation

​

20

    • Criteria that contributes to the selection of Research Ideas and Proposals.
    • Reinforces efforts for Researchers to work with Project Champions

Communication

    • We have established a methodology to evaluate our selection process.
    • The results of this analysis has provoked discussions to potentially adjust one of the Research Idea criterion “Tangible Benefits.”

Evaluation

21 of 22

Next Steps

21

Evaluate complete Proposal Criteria

Evaluate Criteria based on Subcommittee (Five Subject Areas)

Evaluate Authored Research Ideas (NCDOT vs. University)

22 of 22

Questions

Curtis T. Bradley, Ph.D.

E-mail: cbradley8@ncdot.gov

Phone: 919-707-6664

22

This Photo by Unknown Author is licensed under CC BY-SA