Alan Lupatini

António Galvão

Carmelina Mbesso

Rui Parreira

House

Targaryen

Contents

1. Project Intro
2. Datasets and clean-up
3. First Observations: Client Demographics
4. Detailed analysis: Test Performance
5. Conclusions and Recommendations

Project kpis

Who are we?

methodology

What is the experiment?

introduction

Customer Experience team at Vanguard, working on a new experiment to increase digital sales in our main landing page funnel.

Conversion Rate (CR)

Error Rate

Avg. Time Spent per Page

Test Confidence Level

Data is captured using a third-party web analytics software. It is enriched with back end data regarding user demographics. It is all unified and cleaned out using Python. Some reports are generated using SQL and visualized with Python, while some were created with Tableau, depending on convenience.

The conversion funnel was optimized and all pages in the funnel were changed. Management has defined that only a 5% or bigger increase in the Conversion Rate would be considered as a successful experiment.

Client profiles

Experiment Roster

Datasets

Digital footprints

Calculated metrics

Client ID

Tenure (Years)

Tenure (Months)

Age

Gender

Number of Accounts

Balance

Calls 6 Months

Logons 6 Months

Client ID

Variation (Control or Test)

Variation (Page Type)

Process Step

Visits

Unique Visitors

Time Stamp

Conversion Rate (CR) > Amount of Starters that reached Confirmation page.

Error Rate > The rate in which users loaded the same page more than once or navigated back into the funnel.

Time Spent on Page > The amount of time lapsed between loading a page into the funnel and the next page load (unavailable in last page seen in funnel).

Confidence Level > Metric that calculates if the traffic and CR are statistically significant.

​

The distribution of genders in client base is relatively equal

User demographics

Most clients are new in the company

Ages in client base are somewhat concentrated in between 23 and 65, especially between 23-37 and 47-64

Men have, on average, bigger account balances

User demographics

oLDER CLIENTS WITH HIGHER TENURE ALSO TEND TO HAVE BIGGER BALANCES

Since clients with tenure <6 years are majority, they also sum the absolute bigger account balance in comparison to other groups

User demographics

clients with tenure smaller than 6 years are majority

USER BEHAVIOR

To improve conversion rate in 5%

Not increasing process time

Key performance indicator

Kpi analysis: conversion rate & error rate

Test version has 30% less error rate

​

Test version has a lift of 8.4%

With 0.99 p-value, we fail to reject the null hypothesis

(improvement is greater than 5%)

Test version Wins

Kpi analysis: Average time spent

both versions have a total process time of around 173 seconds

Converted users took 10 seconds less, on average, in test version

Test & control version have similar processing times

Now let’s have a look in

our tableau story for other insights

Click to open Tableau Link

​

Challenges & learnings

• Data cleaning, combining data.
• Data combination had to be done again in Python so we could import one single table to Tableau.
• Translating some calculations from Python to Tableau wasn’t easy.
• Calculating test confidence levels on SQL was really difficult, so we turned to Python.

Team work

• Team messaging in Slack
• Tasks distribution in Trello
• We tried maximising individual parallel work when possible

Implement test version

Implement error tracking so there is less guess work related to it

Conclusions

Further improve pages in funnel performing sub optimally

House Targaryen thanks you

Alan Lupatini

António Galvão

Carmelina Mbesso

Rui Parreira

​

Presentation Structure

Feel free to present the project in as you feel best represents your work, but below you will find a suggested format for the presentation as a guide.

1. Title Slide (1 slide):
• Your project’s title and your team members' names.
2. Introduction (1 slide):
• Briefly introduce Vanguard and the context of the digital challenge.
• State the main question: Did the new UI lead to higher completion rates?
3. Data Overview (1-2 slides):
• Present an overview of the three datasets used: Client Profiles, Digital Footprints, and Experiment Roster.
• Outline the data cleaning and merging process you performed.
4. Exploratory Data Analysis (EDA) (2-3 slides):
• Highlight the key demographics and behaviors of Vanguard’s online clients.
• Share initial findings about client engagement before diving into the A/B test results.
5. Performance Metrics (2-3 slides):
• Define the KPIs you chose to evaluate the new design’s performance.
• Compare the KPIs for the Control Group vs. the Test Group.
• Present visual aids to support the KPI analysis.
6. Hypothesis Testing (2-3 slides):
• Discuss the hypothesis tests conducted, the chosen significance levels, and the results.
• Include the tests for completion rate differences and cost-effectiveness threshold evaluation.
• Present any additional hypothesis tests conducted and their outcomes.
7. Experiment Evaluation (1-2 slides):
• Evaluate the experiment design, discussing the randomization, duration, and potential biases.
• Suggest what additional data could have been beneficial.
8. Tableau Visualizations (2-3 slides):
• Showcase the Tableau dashboard and visualizations created.
• Demonstrate how viewers can interact with the data based on demographics.
9. Teamwork & Project Management (1 slide):
• Reflect on how you divided and conquered the project tasks using your Kanban board.
• Share insights about your team’s collaboration and project management strategies.
10. Challenges & Learnings (1-2 slides):
• Highlight any significant challenges you faced during the analysis.
• Discuss how you overcame them and what learnings you took away.
11. Conclusion (1-2 slides):
• Summarize the key findings from your analysis.
• State your final verdict on the new digital interface's effectiveness.
• Offer recommendations for Vanguard based on your insights

​

​

Day 1/2 conclusions

1 - Who are the primary clients using this online process?

• Middle age group (30-50) are the primary clients using the online process, with >4 logons in the last 6 months, if we look at the >16 tenure group.
• We’ve noted that all demographic groups, and all tenure group made at least 3 logons in the last 6 months.
• The same applies to frequency of contact in the last six months, with at least 3 mean calls. Only middle age with >16 years reached 4 mean calls.

​

2 - Are the primary clients younger or older, new or long-standing?

2 .1 Looking at average account balance by age, we can clearly see that seniors (>50 years) are the primary clients peaking at 300.000.

Male clients also almost double female clients, in this analysis, with a peak of 170.000.

​

2. 2 Analysing the tenure group, Seniors with < 6 years have the highest total account balance, but the same applies to middle age and young group.

In the average account balance, we can see that long stand clients (>16 years) have the highest value in the middle age group. Senior comes second, but <16 years slightly overcomes clients with >16 years.

​

​

The test version was superior by a long margin and our statistical calculations indicate that the test should be validated and implemented to all users.

The test version had less errors than the control version, once again enforcing the narrative that the test version is superior and should be the version implemented to all users.