Chen Qingwen

The importance of AB-Test (1)

Target

• A/B Test
• It is a data-driven method widely used in well-known enterprises
• Used to compare two versions (A and B) [products, marketing activities, web design, application]
• To determine which version is better
• Comparison objectives, for example:
• Conversion rate,
• User participation
• income

Example 1

• Compare
• 2 plans
• Web Design
• Which one
• is better

Example 2

• Compare 2 solutions
• 【Whether to add the countdown time】
• Which one is better

Example 3

• Compare 2 page design plans
• 【Whether the model's picture should be enlarged】
• Which one is better

Example 4

• Compare 2 page design plans
• 【Promotional effect of male/female models】
• Which one is better

Example 5

• Compare 2 page design plans
• [Whether you want real-life models to show the effect well]
• Which one is better

Basic concepts of A/B testing

Control Group

Experimental group, Variant Group

Basic concepts of A/B testing

1. Variable setting (A vs. B)
1. Group A (Control Group): Older version, usually existing version, represents an unmodified state.
2. Group B (Experimental Group, Variant Group): New version, including certain changes (such as buttons with different colors, new titles, different ad designs, etc.)

​

​

A/B test evaluation

Key indicators of evaluation

A/B test evaluation Comparison of key indicators

1. A/B test evaluation
• Randomly assign users to pages with different experiences.
• Then calculate the key metrics (such as conversion rate, comparing who has a higher conversion rate)
2. Key indicators:
• Watch ads for stay time
• Add to cart times
• Number of purchases
• Conversion rate

​

​

Conversion rate of marketing funnel

• Marketing funnel model
• The Marketing Funnel

Evaluate the position of the indicator in the funnel

Evaluate the position of the indicator in the funnel

• Calculation of conversion rate (dividing the number of people between every 2 stages)

The situation of famous enterprises and large international manufacturers using A/B

The situation of using A/B by well-known enterprises and large international manufacturers

• How Microsoft Bing uses A/B testing or online experiments to optimize products
• According to Harvard Business Review (HBR) September 2017 article
• Microsoft's Bing performs dozens of A/B test improvements every month.
• The average increase conversion rate is 10%~25%.

​

• This not only demonstrates the practical application of A/B testing in search engines,
• It also highlights the key role of data-driven approaches to commercial success

​

The situation of using A/B by well-known enterprises and large international manufacturers

• International manufacturers:
• Microsoft, Amazon, Facebook, Google
• Widely used A/B test:
• Through experiments, the results of different A/B solutions are verified.
• In turn, increase the conversion rate.
• Test scope covers:
• Adjustment of the user interface (UI) of the screen design,
• Optimization of search engine algorithms,
• Optimization of advertising algorithms,
• Strengthening of personalized recommendation system

​

1. Google – Search results and ad optimization

• Application scope:
• Google uses A/B testing to optimize [Search results pages, ad placement and user experience]
• Through A/B testing, Google will compare the impact of [different search ranking algorithms, page designs and ad formats] on user behavior.
• Case:
• Google has tested 41 different blue hyperlink colors and finally chose the one with the highest conversion rate, which is reported to earn additional millions of dollars a year.
• Google Ads' bidding mechanism also uses A/B testing to adjust different ad display methods to improve click-through rate (CTR)

​

2. Amazon – E-commerce conversion rate and recommendation system

• Application scope:
• Amazon uses A/B testing to [optimize product pages, shopping process, price settings, recommendation algorithms].
• Case:
• The "One-click Buy (1-Click)" button is a feature launched after a large number of A/B tests, reducing the steps of the checkout process and improving conversion rates.
• Amazon also optimizes the "customer buys" product recommendation system through A/B testing to improve cross-selling.

​

3. Facebook – Interface design and user interaction

• Application scope:
• Facebook will A/B test the [different UI/UX] changes to ensure that new features have a positive impact on the user experience.
• For example: Test [different news sorting methods] to improve user participation.
• Case:
• When Facebook developed the "Like" button, it compared the interaction rates of different versions through A/B testing, and finally decided on the design that is most popular among users.
• The launch of "Reactions" has also undergone a lot of A/B testing to ensure that it can improve user engagement more than the simple "like" button.

​

5. Airbnb – Order Conversion Rate and User Trust Mechanism

• Application scope:
• Airbnb improves the conversion rate of property through A/B testing, such as [photo quality, evaluation system, landlord trust], etc.
• Case:
• Improved property photo quality: Airbnb tested whether listing photos taken by professional photographers will increase booking rates, and the results showed that professional photos listings are 40% higher than those uploaded by ordinary users.
• Price recommendation: Airbnb tests dynamic pricing systems, recommended to landlords for the most competitive prices to increase booking rates.

​

A/B testing has become the core strategy for these well-known companies

A/B testing has become the core strategy for these well-known companies

• These cases show that A/B testing can help enterprises continue to optimize products, improve conversion rates, and user satisfaction in data-driven decision-making
• Improve user experience (Facebook, Spotify, Netflix)
• Optimize sales conversion rate (Amazon, Airbnb)
• Improve ads and recommendation systems (Google, Amazon, Netflix)
• Improve APP usage and interaction (Uber, Facebook)

​

In which areas of enterprise management have been used A/B testing?

1. Digital Marketing

• A/B tests are widely used in the field of digital marketing to improve [advertising effectiveness, user conversion rate, interaction rate]
• Application scenarios:
• Email Marketing
• Test: Different titles, content, CTA (Call-to-Action) buttons, and sending time, the impact on the opening rate and click-through rate.
• Online Ads
• Test on platforms such as Google Ads, Facebook Ads and other platforms: find out the best ad version with different ad titles, pictures, and copywriting.
• Social Media Marketing
• Test: The impact of different post formats (plain text, pictures, videos) on user participation (like, share, and comments).

​

2. Finance & Insurance

• A/B testing is mainly used in the financial and insurance industry to improve [customer conversion rate, optimize pricing strategies, and improve risk management efficiency]
• Application scenarios:
• Credit Card vs. Loan Application: Test: The impact of different credit card promotion programs (such as cash return vs. travel points) on the number of applications.
• Insurance product recommendation: Test: Whether different insurance product introduction methods can increase the insurance rate.
• Fraud detection: Test: Whether different risk control mechanisms can effectively reduce the risk of financial fraud

​

3. HR & Recruitment

• A/B tests can be used to improve [job seeker experience, optimize talent recruitment process, and improve internal employee participation]
• Application scenarios:
• Job Search Website Optimization: Test: The impact of different resume templates or recommendation mechanisms on job search success rate.
• Company internal training: Test: Online vs. Offline training courses have an impact on employee learning effectiveness.
• Employee Benefits: Test: The impact of different reward systems (such as performance bonus vs. extra holidays) on employee satisfaction.

​

4. E-commerce (E-commerce)

• A/B testing can help
• E-commerce platform optimizes product display methods,
• Promotions and shopping procedures,
• To improve sales conversion rate.
• Application scenarios:
• Product page optimization: Test: [Different product pictures, descriptions, and pricing methods] impact on purchasing decisions.
• Price and Discount Strategy: Test: [Different discount methods (full discount vs. Percentage discount)] impact on consumer purchasing behavior.
• Checkout process optimization: Test: [Whether to simplify the checkout process (providing one-click checkout, Apple Pay, Google Pay)] can improve conversion rate

​

5. Website and App Optimization (Web & App Optimization)

• A/B testing can help companies:
• Optimize website design,
• Improve user experience (UX)
• Improve conversion rate.
• Application scenarios:
• Landing Pages: Test: [Different button colors, titles, pictures, layouts] on click-through rate and conversion rate.
• Website navigation and user experience (UX): Test: [Different website structures, search functions, shopping processes] impact on user retention.
• APP interface (Mobile Apps): Test: [Different UI designs and function placement locations] to improve user operation experience and retention rate

​

6. Gaming Industry

• A/B testing helps game developers:
• Improve player experience,
• Increase retention and revenue.
• Application scenarios:
• In-game Purchases: Test: [Different prop prices and promotion methods] impact on player consumption behavior.
• Game difficulty adjustment: Test: [Level design (enemy strength, reward mechanism)] whether it affects the player's gaming experience and retention rate.
• Advertising display: Test: [Frequency and type of interstitial ads] to avoid affecting the player's experience while increasing profits.

​

7. Education and Online Learning (EdTech)

• A/B testing is available in the field of education to:
• Improve student participation,
• Optimize the learning platform
• Improve course conversion rate.
• Application scenarios:
• Course recommendation: Test: [Different course recommendation mechanisms (such as AI recommendation vs. manual recommendation)] Effect on student registration rate.
• How to present learning content: Test: [Video vs. Interactive Simulation] The impact on students' learning effects.
• Quiz and evaluation: Test: [Different test modes (multiple-choice questions vs. practical questions)] Can better improve learning effectiveness?

​

Misconceptions about key indicators

Comparison of A/B tests of 2 page design solutions Results: 5,000 customers were tested each, and the number of purchases on the right was high.

• [Is the live model better to show the effect? 】
• Which one is better
• Is the same going forward?

​

Is it better to be on the right in the future?

Is it better to be on the right in the future?

• If you want to ensure that [it is also better on the right] in the future, the premise is:
• It must be statistically: the samples that meet the test [Population] are all [higher on the right]
• Because it is the same as the result of the Population
• You have to do [statistical verification]
• [Statistical Test] is passed, and it is guaranteed to meet this conclusion on [Population]
• Then, future tests will successfully achieve the expected goals

Is it better to be on the right in the future?

• If the verification has not been made,
• Just look at the results of this experiment
• 1200 times on the right > 1100 times on the left
• Only guarantee that the result of this small sample is like this
• but
• The future cannot be guaranteed
• unless
• Pass, testing

Is it better to be on the right in the future?

• Conclusion of the paper of science and engineering students:
• Many are incorrect
• because
• They're just the result of this experiment
• It doesn't mean that this is the case in the future
• Unless they have a testing
• but
• Almost most scientific and engineering papers have not been tested

• Small sample experiment
• Cannot represent the poulation (future)

​

• Because there will be many errors in it, resulting in the current results

​

Correct A/B test

Implementation steps

Correct A/B test implementation steps

Correct A/B test implementation steps

• 1. Clarify the test goals (such as improving conversion rate and click-through rate)
• 2. Select variables and test plans (such as button color, title copy)
• 3. Determine sample size and grouping (ensure randomness and statistical significance)
• 4. Select the test tool (Google Optimize, Optimizely, etc.)
• 5. Perform tests (ensure that the test time is sufficient and avoid interference factors)
• 6. Analyze the test results (check statistical significance to ensure the data is reliable)
• 7. Perform the best version based on the results (continuous optimization, improve performance)

​

【example】

A/B test implementation steps

Give an example

• context:
• An e-commerce website hopes to increase the "shopping cart conversion rate".
• That is, let more users click the "Add to Cart" button from the product page.
• In turn, increase sales.

​

Step 1. Define the test objectives

• Step 1. Define the test objectives
• Increase the click-through rate (CTR) of the "Add to Cart" button.
• In turn, increase the shopping conversion rate (Conversion Rate)

​

2. Select variables and test plans

• Test variables:
• Test whether the color of the Add to Cart button affects click-through rate.

​

• A/B Test Solution:
• Version A (Control group): The button color is blue (original design).
• Version B (mutation group): Change the button color to orange (assuming orange can attract more attention)

​

3. Propose [Hypothesis]

• Hypothesis:
• If the button color is changed from blue to orange,
• Users are more likely to notice the "Add to Cart" button, and the click rate is expected to increase

4. Determine sample size and grouping

• Sample size calculation:
• Use A/B test sample calculation tools (such as Optimizely, Google Optimize) to calculate the number of samples.
• Based on current traffic, it is determined that the test requires at least 20,000 visitors to achieve statistical significance.
• Random grouping:
• 50% Visitors See Version A (Blue Button).
• 50% Visitors See Version B (Orange Button).
• Statistical significance criteria:
• Set Confidence Level 95% (p value < 0.05).

​

5. Select the test tool

• Test platform:
• Google Optimize (free, suitable for website testing).
• Optimizely (advanced version, more variables can be set).
• VWO (Visual Website Optimizer) (Can track user behavior).
• Data tracking method:
• Use Google Analytics (GA4) to record click-through rate changes.
• Monitor changes in "Car Conversion Rate".

​

6. Perform tests

• Test time:
• Set the test time for 2 weeks to ensure the data is stable.
• Ensure that the test process is not disturbed:
• No other website revisions or promotions will be carried out to avoid affecting the test data.
• Period monitoring:
• Check data changes daily to ensure there are no technical errors (such as buttons that cannot be clicked or tracked failed).

​

7. Analyze the test results

• After the test is completed, analyze the key indicators:
• Version A (blue): Click-through rate 5.2%
• Version B (Orange): Click-through rate 7.8%
• Statistical significance detection (p value < 0.05):
• The click-through rate of the button in version B is significantly higher than that in version A.
• The conversion rate (cart conversion rate) also increased by 6%.

8. Conclusion

• in conclusion:
• The orange button is more attractive to users than the blue button.
• And the data reached statistical significance (p < 0.05)

​

【CP value】

The value of this course

【CP Value】The value of this course

• A/B test is a technology used by seniors from the department to do marketing in the industry. I asked the teacher.
• But the current teaching of A/B testing,
• All of them only talk about concepts, no practice
• Most of them are foreign language teaching
• SPSS example teaching not found
• There are few other methods for teaching actual A/B test examples

【CP Value】The value of this course

• Features of A/B testing teaching in this course:
• Talk about concepts and practice
• Chinese teaching
• Three ways of implementation:
• SPSS software operation (a rare SPSS example at home and abroad)
• Python program analysis
• ChatGPT Quantitative Analysis
• Multiple actual A/B test examples

Ranking of CP values ​​taught by teachers

• 1. A/B test implementation (rare SPSS examples, three implementation methods)
• 2. Senior Year: Customer Relationship Management
• 3. The third triumph: Internet marketing
• 3. Sophomore year: Application of artificial intelligence in business management (Pandas business data analysis)
• 4. Sophomore year: Database Management and Application (Mysql Database Analysis)
• 5. Junior Three: Application of Artificial Intelligence in Business Information Prediction (AI Machine Learning, Deep Learning, Business Information Prediction)
• 6. Become a bigger one: Python programming

​

​

​

​

The teacher teaches at the Institute of Peking University of Science and Technology with [DDDM, data-driven decision-making]

Related series of courses

Very popular now

【Data-driven decision-making model】

DDDM

Data-Driven Decision Making

Data-driven decision-making, DDDM

• Data-Driven Decision Making (DDDM)
• It is the [business model and decision-making method] that uses data as the core basis

​

• When a company or organization makes decisions,
• Mainly based on the analysis results from various internal and external data,
• Instead of relying solely on intuition and experience to make decisions

​

The teacher taught a series of courses related to [DDDM, Data-driven Decision Making] at the Institute of Peking University

• 1. Market separation and differentiation marketing: questionnaire and SPSS analysis
• 2. E-commerce website traffic analysis (Google Analytics, GTM)
• 3. Current situation analysis: SQL data analysis of transaction data
• 4. Current situation analysis: Pandas data analysis of transaction data
• 5. Machine learning models are applied to business situation prediction (Python machine learning)
• 6. Data visualization and business intelligence analysis (Tableau, Power BI, Google Looker Studio)
• 7. RFM model (customer group analysis, and customer value prediction)
• 8. What is the automatic product recommendation system for e-commerce? ? ?
• 9. Social Media Data Analysis (Meta, IG…)
• 10. A/B Testing and Data Experiment
• 11. Use [Commercial CRM Customer Relationship Management System] for data analysis

• The End

​