Online News Popularity Prediction

Anuradha Boche

Introduction

• Set of features about online articles published by Mashable over a period of two years (c. 2015).
• Goal is to predict whether an article is popular or not based on number of shares.
• Number of attributes: 61 (58 predictive attributes, 2 non-predictive, 1 goal field)
• Number of rows: 39797

Types of features:

• Words: Words in title / content; Rate of unique / non-stop words
• Digital Media: No. of links to other articles in Mashable; No. of images / videos
• Keywords: No. of keywords; Article category
• NLP: Closeness to 5 LDA topics; Title / content polarity
• Target: Number of shares at Mashable

Dependent variable - Number of shares

Without transformation (non centered, non linear and right skewed)

​

​

​

​

With log transformation (more centered, linear and normally distributed)

​

Feature selection

• First, performed logistic regression on all variables (58)
• Using above fit, performed stepwise logistic regression to select best subset of variables (36)
• Applied Boruta algorithm on the best subset of variables to obtain our final subset of variables (32)
• Boruta is a feature selection algorithm based on Random Forest which creates shadow features
• It then randomly shuffles values for each feature to remove correlation and compares them to original features for importance
• PCA was not used since that results in degrading the model results

Important predictors

Independent variables (Predictors)

Weak Predictors (Categorized as tentative by Boruta and low importance in RF)

1. Title subjectivity and absolute title subjectivity
2. Number of words in the title
3. Article type is lifestyle or business
4. Polarity of article

Little or no relationship (Very high p-value > 0.5)

1. Weekday is Monday, Tuesday etc
2. LDA_04 has perfect collinearity with other variables (singularity)
3. Number of images or videos
4. Article type is world

Lastly, when we run lasso regression we can observe that features like article type is world, no. of videos, article polarity etc have been regularized by shrinking their coefficients to 0.

Relationship b/w predictors

Data channel vs LDA topics

​

​

​

​

​

​

• The dataset does not provide any LDA topic information
• Looking at the average value of LDA (lower is closer) we can see for example lifestyle articles are similar to LDA1, entertainment articles to LDA4, technology articles to LDA3 etc.

Categorical predictors

Type of article vs Popularity

Publishing day vs Popularity

Relationship with important predictors

Relationship with important predictors

Relationship b/w important predictors

Model ROCs

Area under the curve: 70.19%

Area under the curve: 66.17%

Area under the curve: 64.17%

Model scorecard

Recommendation

Traits for popular articles:

1. Articles posted on the weekend are more popular
2. Articles based on social media get more shares while articles based on entertainment get fewer shares
3. Similarly, articles close to LDA02 get more shares while articles based on LDA00 get fewer shares

Future work:

1. Work on creating NLP features ourselves to better understand what each topic means, essentially working with all words in the articles
2. Work on data cleaning for the features in depth to create a better explained model

​

​

Thank You! Questions?

Appendix

Appendix