Advanced AI

Final Report

資工碩一

R09922188 曾泓硯

Outline

1. Project Proposal
1. Team Members
2. Motivations
3. What is the state of the art?
2. Introduction
• Look into the data
• Evaluation
• Problem Definition
3. Classification
• Data Preprocessing
• Experiment setting
• Results

4. Regression
1. Data Preprocessing
2. Experiment setting
3. Results
5. Ensemble
• How to do it?
• Results
6. Failure Attemptions
• Methods & Results
7. Conclusion

​

Project Proposal

​

1

Team Members

ID/Name：R09922188 曾泓硯

Education： NTHU EE -> NTU CS

Research Topic： Deep Learning on HDR tone mapping

Project Experience related to AI：

Adaptive Learning in Education, Facial Super-Resolution,

Lensless device & Blind-deblurring, Style Transfer on Android Devices

Motivation/Background

1. Advertisement is now everywhere, which is annoying.
2. I hope to get useful information only.

What is the state of the art?

• The setting is quite different with the original learn to rank, which label is in general binary.
• There is a learn to rank challenge hold by Yahoo in 2011, and the best team get the NDCG for about 0.8. However, the definition of NDCG in this contest is different with original ones.
• There are two difference, the numerator of NDCG is the truth amount of consumption and we only choose the top 3 categories to calculate the NDCG.

Introduction

2

Look into the data

Evaluation

Problem Definition

1. Multiclass classification
1. Given the data of one customer, predict the probability of each shop tag that the customers are most likely to spend on in the next month.
2. Given the data of one customer, predict the shop tags for 2nd, 3rd most likely to spend on in the next month.
2. Regression
• Given the data of one shop tag, predict the total amount of consumption in the next month.
3. Ranking
• Given a set of data according to each target shop tags of an customer, rank the data for the next month.

Classification

3

Data Preprocessing

• Consider appropriate constant for imputing missing data.
• Some of the data values exceed the max value of float32, thus I apply logarithm function to scale down the range.
• Merge all of the tags within one month for a customer into one row.
• Some of the columns are related, in order to normalize the data into the range between 0~1, I divide one column with another. (ex. tag1_amt/ txn_amt)

Data Preprocessing

Experiment setting

1. Assume the amount of consumption is proportional to the probability of classification results.
1. Merge the data by each customer from month 1 to month 12 as input. Sum the amount of consumption of each tag from month 13 to month 24 and take the max label as ground truth.
2. View each rank as an individual multiclass classification and train 3 models for each rank.
3. Assume the time when the data is collected has different effect.
• Trained 23 models to predict the result of next month and merged the probability together.
• I expect that the importance of data may be affected by the time of when the data is collected. Therefore, I ensemble these models with weighted sum according to the time.

Results

# N: Setting N from previous slide

Feature Importance

Regression

4

Data Preprocessing

• Consider appropriate constant for imputing missing data.
• Use scikit-learn packages to perform scalar transformation, instead of dividing by related column.
• Merge all of the month data within a tag for a customer into one row.

Data Preprocessing

Experiment setting

1. Assume the consumption trend of each shop tag is consistent with the sum of all shop tag.
1. Merge all of the data together, and predict the total amount of consumption next month.
2. Assume the consumption trend of each shop tag is different to the sum of all shop tag.
• Training on the whole data of shop tag, and predict the amount of consumption in each tag next month.

Results

Feature Importance (XGBoost)

Ensemble

5

How to Do it?

1. During Training (Regression as example)
1. Separate training data into two parts, data of predicted tag/ not predicted tag
2. Train two model respectively, and sum the result from both model
2. Ensemble the results
• Ensemble from previous result, including classification, regression, ranking.
• Weight the top 3 label with [3,2,1] of each previous result and sum up to get the final prediction.

Results

Failure Cases

6

Methods & Results

1. Deep Learning (Linear Model: For classification)
2. Learn to Rank (LambdaMART : Based on Regression Methods)

Conclusions

7

Classification

1. Potential Problem:
1. Merging all of the data together during training may affect the performance of model.
2. How to combine the month information can be done more systematically by models.
3. Classification results isn’t exactly what we want.
2. Observations:
• Using the probability to rank has the same effect as using 3 models.
• Weighting result from different month may improve the performance.
• The feature related to the predicted tag, whether buy in offline/ online, domestic/ oversea might affect the prediction results.

Regression

• Potential Problem:
• Some of the regression ground truth is 0, whether we should remove the data is questionable.
• Regression results isn’t exactly what we want, but better than classification results.
• Observations:
• Merging all of the data together during training may affect the performance of model.
• Training with data from predicted tag is enough to get decent results.
• XGBoost model cares more about data from recent 11 months, whether buy in offline/ online, domestic/ oversea and personal information.

Ensemble & Failure Cases

1. Ensemble methods doesn’t perform better than origin model.
2. Deep Learning might still be able to solve the problem of classification, but I failed with naive attemptions.
3. Ranking model fail to outperform other models, which is a surprising fact.
4. Sparse data may still be able to get decent results.

Next Step?

1. Learn to downsample the label to deal with unbalanced data
2. Apply deep learning method on regression

Reference

[1] Olivier Chapelle, Yi Chang, Yahoo learn to rank challenge

[2] Chen, T., & Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (pp. 785–794). New York, NY, USA: ACM. https://doi.org/10.1145/2939672.2939785

[3] Scikit-learn: Machine Learning in Python, Pedregosa et al., JMLR 12, pp. 2825-2830, 2011.

[4] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., … Chintala, S. (2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32 (pp. 8024–8035). Curran Associates, Inc. Retrieved from http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

Q&A

1. 我後來有做一個版本只取 13~23 月份的資料，但效果也是差不多。