1 of 13

Introduction to Data Science

Final Project

​

House Prices - Advanced Regression Techniques

第20組 章元豪 楊哲睿 李玟卉

​

2 of 13

Outline

  • Introduction
  • Data Processing
  • Feature Selection
  • Model
  • Result
  • Conclusion

​

3 of 13

Introduction

  • House prices prediction competition from Kaggle
  • Predicted the final price of each house in Ames, Iowa
  • Train data and test data contain 79 house features

4 of 13

Data Processing

  1. Fill missing values

​

​

​

​

  1. Dealing with outliers

​

5 of 13

  1. Ordinal Feature ——> Label Encoding ——> pd.map() Nominal Feature ——> One-Hot Encoding ——> get_dummies()

​

  1. Standardization

6 of 13

Feature Selection

Method 1：Select all features

Method 2：Select features with a correlation coefficient more than 0.5 and remove features that are highly correlated with each other

(next slide)

7 of 13

From the figures, you can see how much each feature is related to Saleprice and how closely related features are highly related to Saleprice.

Conclusion：

  1. OverQual is the most highly relevant with 0.79.
  2. Choose the feature with a higher correlation with saleprice if the two features with a correlation of more than 0.8 and delete the other.
  3. Delete GarageArea、1stFlrSF、TotRmsAbvGrd feature

8 of 13

Model

  • Lasso()
  • ElasticNet()
  • XGBRegressor(objective='reg:squarederror')
  • GradientBoostingRegressor()
  • LGBMRegressor()
  • BayesianRidge()

​

9 of 13

  • Method 1 (RMSE)

Lasso : 0.1755 (0.0158)

ElasticNet : 0.1678 (0.0152)

XGBoost : 0.1325 (0.0112)

GBoostRegressor : 0.1289 (0.0128)

LightGBM : 0.1306 (0.0104)

BayesianRidge : 0.1235 (0.0175)

  • Method 2 (RMSE)

Lasso : 0.2322 (0.0083)

ElasticNet : 0.2277 (0.0074)

XGBoost : 0.1656 (0.0110)

GBoostRegressor : 0.1650 (0.0123)

LightGBM : 0.1710 (0.0146)

BayesianRidge : 0.1709 (0.0088)

10 of 13

Result on Kaggle

  • Submit top 2 models of these two methods to Kaggle
  • BayesianRidge：1039/4626(22%)

11 of 13

Best result on Kaggle

  • Adjust the parameters of the GBoostRegressor model

GBoost = GradientBoostingRegressor(n_estimators=3000, learning_rate=0.05,

　 max_depth=4, max_features='sqrt',

　min_samples_leaf=15, min_samples_split=10,

　 loss='huber', random_state =5)

  • GBoostRegressor：641/4626(14%)

12 of 13

Conclusion

  1. Selecting features with a correlation coefficient more than 0.5 may miss the hidden key features.
  2. We may use ensemble learning to train the base model by combining multiple weak learners.
  3. Do the tuning of XGBoost parameters.

13 of 13

Thanks