Predicting Success in Training of Navy Aviators

Neil C. Rowe and Arijit Das

Computer Science Department�U.S. Naval Postgraduate School

http://faculty.nps.edu/ncrowe

1

The challenge of Navy aviation training

• Navy training of pilots and flight officers (“aviators”) is expensive:
• The equipment needed is expensive.
• Training is time-consuming since equipment is complex.
• When someone leaves the training program, considerable money has been wasted.
• Prediction of training performance is difficult:
• Hundreds of factors have been tested, but no factor is dominant.
• Curricula based on the type of aircraft and the trainee role. Different curricula get different tests, so it is hard to compare candidates.
• Skills decay is significant with aviation training.

​

​

​

​

​

2

Data used in our study

• 143 Excel tables from U.S. Navy Fleet Forces Command, with many entries blank
• 18,596 training candidates, both pilots and flight officers
• 301 metrics on the candidates:
• Standardized tests given early in the training
• Averaged tests on coursework for particular curricula
• Averaged flight tests for particular curricula
• Background data like race, organization, and flight school
• Counts on the number of non-null attribute values for each phase
• Pilot names and personal information were excluded from the data.

3

Cleaning the data

• Data preponderantly numeric was converted into numbers.
• Data that was preponderantly nonnumeric was all converted into symbol values.
• Where reasonable, nonnumeric sortable values were converted to numbers. For instance, “Complete”, “Pass”, “Incomplete”, “Conditional Pass”, and “Pass” were converted to 1, 1, 0.5, 0.5, and 0.
• Dates were converted to epoch time integers (seconds since January 1, 1970 at midnight) to be easier compare.
• Null values were frequent in the data:
• The empty string, a string consisting of a single space, “N/A”, “#N/A”, “NONE”, and “NULL” were all standardized as “null”.
• 4804 null values for candidate ID codes occurred from 2000 to 2010; they were replaced with consecutive negative numbers since the rest of their data had significant information.
• Null values for numeric attributes generally meant missing data, so we excluded them from averages.
• Nulls for nonnumeric attributes were generally important, such as a null for previous flight training meaning the candidate had none.

​

4

Joining the data

• Missing and redundant data created obstacles to prediction of candidate performance.
• Fix 1: We averaged all scores for a candidate on the same test metric.
• Fix 2: We averaged those averages for a candidate within a curriculum, dividing by the difficulty level on flight tests.
• Fix 3: We did “outer joins” instead of the usual “inner joins”. That meant we entered nulls for attribute values that were missing for candidates.

​

​

​

5

Respecting attribute order

• Each attribute is associated with one of 11 phases of training:
• PRE: Initial data before the candidate begins training
• ASTB, Aviation Selection Test Battery: Initial testing
• IFS, Initial Flight Screening: Initial flight school
• API, Aviation Preflight Indoctrination: Classroom instruction
• PRI, PRI1, PRI2, Primary Flight Training: In specialized curricula
• INT, Intermediate Flight Training: In specialized curricula
• ADV, ADVCORE, Advanced Flight Training: In specialized curricula
• FRS: Fleet Replacement Training: Refresher training
• Training follows the order:
• PRE – ASTB – IFS – API – PRI – PRI1 -- PRI2 -- INT – ADVCORE -- ADV – FRS
• It only makes sense to predict attribute values from those of previous phases.

6

Methods of predicting attribute values

7

Attributes measuring candidate success

After discussion with the sponsor, we identified 38 possible measures of success of a candidate:

• Counts of retesting or test failure
• Indications of disenrollment, inactive status, or “attriting”
• Counts on data for later phases, suggesting the candidate was disenrolled if zero
• Average academic and flight-test grades

​

8

Significant binary correlations with candidate success

9

Observed correlations

• The reliability of the correlations with success was hampered by the low rate of failure. For instance, IFS_STATUS recorded 13,460 candidates who completed and 374 who were “disenrolled”; SYL_ST had 5369 who completed and 260 who were “attrited”.
• Correlations on later phases did not include candidates who left at earlier phases and who might have provided useful data.
• There were some strong correlations of success with increasing dates, but these are likely spurious due to more complete data for recent candidates.
• Though success correlated with the number of flight hours, it correlated negatively with “formal flight instruction hours”, which may indicate civilian training needed to be unlearned.
• Female gender and minority race showed more failures early in training but fewer later in training.
• Several ASTB test results correlated well with success in IFS, Primary, Intermediate, and FRS, but they were not as helpful in predicting success in Advanced training.
• Several Primary, Intermediate and Advanced training grades correlated positively with both success in Advanced training and FRS. However, some of the advanced-training grades correlated negatively with success, and these should be investigated further.

​

10

Finding good attribute subsets for regressions

• To find more quickly good sets of variables for regressions, a greedy algorithm can assess one variable at a time, deciding whether to include it; that can also consider the number of rows of data with values for the variable, a key issue with our data.
• Additive methods build a regression model by successively adding the variable that improves the fit the most. We preferred an additive approach because of the many missing data values.
• We should only add attributes that:
• Are shown to be correlated with the target variable of the regression;
• Where the number of data rows with nonnull data was less than twice the number of attributes;
• Whose inclusion improves the regression fit by at least 0.1%.
• We applied the greedy algorithm to pick subsets of these input variables, did linear regressions, and measured the standard error.

11

Best regression formulas found (1)

12

Best regression formulas found (2)

13

A database design

14

Conclusions

• We found quite a few factors helpful in predictions, some obvious and some not.
• Some factors measured as significant such as previous flight training, gender, and race are not ones the Navy can control practically or legally.
• Overall, we conclude that the Navy is doing a good job predicting performance of candidate candidates from their multistage testing.
• Currently we are working with Global Technology Inc. in Atlanta to extend this work:
• We will explore the approach of combining factors with a set-covering machine-learning algorithm to find the most useful set of factors.
• We did not have enough data for neural networks, but they should definitely be tried.
• Future work should try to obtain more complete data on the candidates, as we were unable to make many potentially useful comparisons.

​

​

15