1 of 8

AMEX Express Innovation Labs AI Hackathon

2 of 8

American Express Innovation Labs AI Hackathon

2

Feature Engineering

We eliminated features with high number of null values as this indicated insufficient data available:

We conducted empirical testing to determine the optimal threshold of null values that we would use for feature elimination - we trained the model with the different thresholds and evaluated if it affected performance through metrics like accuracy and feature_importance.

Null Values

The ideal threshold was determined to be 9 million.

3 of 8

American Express Innovation Labs AI Hackathon

3

Feature Engineering

We selected features with low correlation between each other and high correlation with the target variables:

Collinearity

We determined that we should eliminate features with more than absolute values of 0.8

High collinearity between each independent variable often leads to:

Overfitting, where the model captures both the underlying trends in the data (signal) and the stochastic error inherent to the dataset (noise)

inflating the standard errors of the regression coefficients, leading to less precise estimates and difficulties in detecting significant predictors

4 of 8

American Express Innovation Labs AI Hackathon

4

Imputation of missing values:

Types of imputation we experimented with:

Mean/Median/Mode Imputation

Linear Interpolation

Best suited when the data are missing completely at random

Best suited when data is not ordered in any manner

Best suited when the data is ordered in a time-series manner

We ultimately used median as it led to higher performance.

Cubic Spline Interpolation

Provides better approximation on data with complex behaviour

May not capture underlying behaviour of data well

Too time-consuming and inefficient for to be practical

5 of 8

American Express Innovation Labs AI Hackathon

5

Our Algorithm

Sampling

Training

Predicted Score Calculation

Testing

Starting with randomly sampling 100 000 data points, we gradually increased to 12,229,978 (size of train dataset) for model training.

We split the features into X and y to predict ind_recommended followed by activation. We used 5-fold sampling to obtain 5 sets of train and test datasets to ensure model reliability.

We trained various ML models to predict ind_recommended based on the selected features.
We then trained a separate instance of the same ML model to predict activation, but including the ground-truth ‘ind_recommended’ column as a feature.

Using the equation: P(activation = 1 | ind_recommended = 1) * P(ind_recommended = 1) - P(activation = 1 | ind_recommended = 0) * P(ind_recommended = 0), we calculated the pred_col.

With the trained models, we first predicted (on the eval dataset) the probabilities of a merchant being recommended to a customer or not (ind_recommended = 0/1).

Then, we predicted (also on eval) the probabilities of a customer activating GIVEN that a merchant was recommended, and GIVEN that a merchant was NOT recommended.

After cleaning our data and selecting the features to use for training, we formulated our algorithm.

First, we started off with random sampling. As the size of the train dataset was very large at about 12 million rows, we started off small, using 100 000 data points to gauge how long the ML models would take to train them. Afterwards, we gradually increased the sample size to obtain more reliable results.

For training of the data, we split the features into X and y to predict ind_recommended followed by activation. At this stage, we used a 5-fold cross validation to obtain 5 sets of train and test datasets. This was to prevent overfitting of data and ensure model reliability and stable accuracy.

For our training algorithm, we first trained various ML models to predict ind_recommneded based on our selected features. �We then trained a separate instance of the same ML model to predict activation, but including the ground-truth ind_recommended column as a feature.

So essentially, we have two trained models, one for predicting ind_recommended, the other for predicting activation.

With these trained models, we predicted the probability of a merchant being recommended to a customer or not on the eval dataset.

We then predicted the probability of a customer activating given that the merchant was recommended to them and the probability of a customer activating given that the merchant was NOT recommended to them.

Using the results we obtained, we calculated the predicted score for each Customer X Merchant pair using the equation,

probability of a customer activating given that the merchant was recommended to them * probability of a merchant being recommended to a customer

- probability of a customer activating given that the merchant was NOT recommended to them * probability of a merchant NOT being recommended to a customer

This equation was derived from the incremental activation rate formula.

and finally, This is how we obtained the dataset we submitted of the three columns: customer, merchant and predicted_score.

6 of 8

American Express Innovation Labs AI Hackathon

6

1

XGBoost

2

Logistic Regression

3

Naive Bayes

4

Random Forest

5

Support Vector Machine (SVM)

Machine Learning Models

Most efficient in handling the large dataset.

With high predictive accuracy, it yields the highest score.

We tested our algorithm with different machine learning models specialised in binary classification.

Due to extensive non-linear relationships in the dataset, it underperformed significantly compared to the other models

SVM was highly inefficient in handling the large dataset, which makes training the model infeasible.

NB assumes that all features are independent of one another, which is not true in this real-world application, hence leading to lower predicted scores.

Random Forest is not as efficient.

This lowered the accuracy of the model in predicting scores.

We ultimately used XGBoost as it was the most efficient and accurate in predicting the activation rate for each customer

A very important step in ensuring our algorithm works and gave the best results was in choosing the correct machine learning model.

As we were predicted ind_recommended and activation, both of which were either 0 or 1, we specifically tested various ML models specialised in binary classification.

We tried a total of 5 models: XGBoost, Logistic regression, naive bayes, random forest and support vector machine (SVM), to explore how each would vary in predicting the scores.

For XGBoost, we found that it was the most efficient in handling the large dataset, taking only a few mins. It also had the highest predictive accuracy and yielded the highest score.

For Logistic regression, it underperformed compared to XGBoost. Logistic regression assumes the linearity between the dependent variables, the ind_recommended and activation, and the independent variables, the selected features. However, due to the nature of the non-linear relationships present in the dataset, LR was not able to produce as high of a score.

For Naive Bayes, it assume that all features were independent of one another. However, in the real-world setting, this is not true. For instance, from our selected features, customer spending is highly dependent on factors like industry trends, their digital activity, and individual preferences

This resulted in lower predicted scores.

Next, for Random Forest, it was not as efficient and prone to overfitting, which significantly lowered the accuracy in predicting the scores.

Lastly, for SVM, it was highly inefficient in handling the large dataset, taking more than 30 minutes for our initial sample size of 100 000. This made the training of the SVM model infeasible.

So, ultimately, we used XGBoost as it was the most efficient and accurate in predicting the activation rate for each customer.

QUESTIONS:

How does XGBoost work?

XGBoost builds a predictive model by combining the predictions of multiple individual models, often decision trees.

The algorithm works by sequentially adding weak learners to the ensemble, with each new learner focusing on correcting the errors made by the existing ones.

It uses a gradient descent optimization technique to minimize a predefined loss function during training.

Why it works so well? -> XGBoost is able to handle complex relationships in data, regularization techniques to prevent overfitting and incorporation of parallel processing for efficient computation

Of the different models, why did you choose these 5?

These models are known for their high performance across a wide range of datasets and problem types.

Each model differ from one another in terms of their interpretations and unique properties.

Logistic regression and Naive Bayes offer straightforward interpretations, which can be crucial in areas where understanding the model's decisions is essential.

Random Forest, XGBoost and SVM are robust to overfitting and noise, making them suitable for a variety of datasets without extensive preprocessing.

While there are other models such as neural networks and deep learning, they often take a much longer time to train than any of the models we have used and don’t particularly add on more to the 5 we have chosen.

What are some non-linear relationships in the dataset?

An example would be customer spend and merchant recommendation.

Linearly, if a merchant is recommended to a customer, it could lead to an increase in customer expenditure at that merchant.

However, the relationship could also be non-linear, as factors like the customer's preferences, the attractiveness of the recommendation (e.g., personalized vs. generic), and the timing of the recommendation might influence spending in complex ways.

Why is SVM so slow?

SVM is slow because of their time complexity. The model involves solving a quadratic optimization problem, which is computationally intensive and requires a longer time if the dataset is large.

SVM is known to be slow to train, why did you insist on using it as a model to explore?

We wanted to use SVM as for binary classification, it classifies the two classes using the best hyperplane that separates all data points of one class from those of the other class. Compared to logistic regression, it is less prone to overfitting and can handle non-linear classification using kernel functions. Hence, we wanted to try and use it with a smaller sample size to find out how its predictive accuracy in determining the scores.

How does Naive Bayes work?

This model predicts the probability of an instance belongs to a class with a given set of feature value. It is a probabilistic classifier. It is because it assumes that one feature in the model is independent of existence of another feature.

It is fast and making prediction is easy with high dimension of data.

How does Logistic Regression work?

LR uses mathematics to find the relationships between two data factors. It then uses this relationship to predict the value of one of those factors based on the other

How does Random Forest work?

Random Forest grows multiple decision trees which are merged together for a more accurate prediction. The logic behind the Random Forest model is that multiple uncorrelated models (the individual decision trees) perform much better as a group than they do alone

7 of 8

American Express Innovation Labs AI Hackathon

7

Generation of pred_col

P(activation given that merchant is recommended) * P(merchant is recommended) -

P(activation given that merchant is NOT recommended) * P(merchant is NOT recommended)

Predicted Score

This approach yielded a higher objective score

Instead of simply predicting the probability of a customer activating given whether the merchant was recommended or not, we decided to include the probability of the merchant being recommended in the first place.

8 of 8

American Express Innovation Labs AI Hackathon

8

Optimisation of objective score

We realised that by giving greater weight to the probability of activation given that a merchant was recommended, the objective score improved significantly.

We achieved this by multiplying the first part of the expression, P(activation = 1 | ind_reco = 1) * P(ind_reco = 1) by some constant alpha.

Then, to determine the optimal value of alpha, we simply performed binary search between our initial guesses of 1.5 ≤ alpha ≤ 10.

The ideal alpha was determined to be approximately 3.00.

At this juncture, we were not quite satisfied with our results yet, and decided to experiment with making adjustments to our objective function formula. Our logical intuition was to give greater weightage to the incremental activation portion of the formula, since we felt that it would maximise the the difference between the test and holdout activation rate. Interestingly enough, our initial guess of multiplying the incremental activation by some constant alpha of 1.5 led to a noticeable increase in the objective score. A second try of alpha = 2 further confirmed this finding.

Of course, we knew that that the objective score would not scale forever as alpha increased. Thus, with limited time on our hands, we decided to perform a binary search on alpha to determine its optimal value. After about 9 iterations, the results were convincing enough to suggest that this optimal value was around 3. At this point, we had run out of submission opportunities. However, we were interested in potentially experimenting with a new constant beta to be applied on the organic activation portion of the formula.