Modeling Faculty/Course Evaluations

Zachary Goldstein and Karan Bokil

The Dataset

• Available for download on cmu.smartevals.com
• Faculty/course evaluation survey responses from Carnegie Mellon University, including:
• Pittsburgh & Qatar Campuses
• Undergraduate and Graduate level courses
• Courses from 2004 to the present.
• Features such as department, course ID, semester, and professor
• Features for ratings such as "Interest in student learning","Explain course requirements", "Clear learning goals", "Overall teaching", "Overall course.”

Data Cleaning

• Could not download singular dataset for all schools
• Merged each individual school’s dataset together
• Some schools had extra fields for evaluation which were removed
• Made new field for which school course is from and which campus it was from (CMU vs Qatar)
• Converted quantitative fields from string to float
• Kept only records from 2010 and above (data before used different fields for evaluation)
• Removed all incomplete records
• Coded categorical features numerically using dummy variables
• Split data into training and testing sets

Exploratory Data Analysis

Visualization of Correlation Matrix

Feature Selection

Due to high pairwise correlations between the variables, just three features are needed to maximize accuracy of model. Which three features are selected from the list of survey questions is less important.

Pittsburgh vs Qatar Feature Selection

• Adding the feature ‘campus’ resulted in a slight improvement in model accuracy.

Without campus

With campus

Course Level Feature Selection

• Adding the course’s level (15-100 vs 15-200) had slight improvements to the model’s accuracy (R-Squared value) as well.

​

Without course’s level

With course’s levels

Model Selection

Source: http://scikit-learn.org/stable/tutorial/machine_learning_map/

​

Evaluating Models and Cross Validation

1. Ordinary Least Squares Linear Regression Model:�Cross Validation R-Squared Values [ 0.81818775 0.81761879 0.86481269 0.84731258 0.79145228 0.85584777� 0.79671685 0.83664137 0.83444244 0.85319176]�Average of Cross Validation Scores: 0.831622427409�--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
2. Lasso Model (when there are two highly-correlated features, picks one at random):�Cross Validation R-Squared Values [ 0.75946583 0.7101192 0.77053785 0.74681557 0.73659003 0.73016895� 0.73016143 0.76169337 0.7383581 0.76090029]�Average of Cross Validation Scores: 0.744481062305�---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3. Elastic Net CV Model (when there are two highly-correlated features, incorporates both):�Cross Validation R-Squared Values [ 0.81837575 0.81729791 0.86468304 0.84707477 0.79216322 0.85518011� 0.79685985 0.83701959 0.83446249 0.85320984]�Average of Cross Validation Scores: 0.831632657869

Conclusions and Implications

• Attributes of good teaching are valued the same amongst Pittsburgh and Qatar students, Undergraduate and Graduate students, and students in different CMU colleges and departments
• Explaining subject material is the variable most correlated with overall teaching quality (important information for teachers to consider), while showing respect for students is the least correlated.
• Even after accounting for all survey questions, some variation in teacher quality (about 15%) remains unexplained, suggesting the need for experimentation with additional survey questions asking about things like an instructor’s energy, excitement, humor, etc.