ARCHITECTING AZURE AI CLOUD PIPELINE TO IMPROVE PRODUCT INFORMATION FOR CUSTOMERS

1

4/5/2024

Industry Partners:

THE TEAM

2

4/5/2024

Harish Datta

Chitneni

Srujana Kalyadapu

Goutham Kumar

Vemasani

Harthik Miriyala

Yu-Hui Lin

Jinxin Ren

Business Problem Statement

Several products lack suitable or missing descriptions that align well with the displayed product leading to:

• Inconsistency in product descriptions across products
• Lower customer satisfaction
• Higher attrition rates

​

​

3

4/5/2024

Missing Product Descriptions On Retailer’s Website

Business Benefits

Including the missing descriptions on the products page will benefit the retailer in the following ways:

4

4/5/2024

Business to Analytics Problem Framing

• Apply LLMs and Image Recognition technology to the analytical process
• Deploy an end-to-end pipeline on Microsoft Azure to help the retailer solve the product description inconsistency automatically at scale.

5

4/5/2024

Constraints:

Subjectivity in Evaluation: The scoring model may introduce subjectivity in assessing factors like readability, relevance, and correctness. Balancing objective criteria with subjective judgment is crucial for fair and consistent assessments.

​

Vendor Engagement: Involving vendors to enhance machine-generated descriptions adds complexity and potential delays. Coordinating with vendors for timely feedback and revisions poses challenges

.

Cost and Resources: Deploying and maintaining the cloud pipeline in Microsoft Azure, training machine learning models, and managing associated costs require resources and expertise. Careful consideration is needed for cost-effective model development and infrastructure maintenance.

​

6

4/5/2024

Datasets

7

4/5/2024

ActiveUPCMarketingInformation.csv

UPCImages.csv

1

*

About the Data

8

4/5/2024

Out of these unique SKUs in the dataset, availability of product descriptions and the corresponding images are as follows:

107,092

3,686

147,714

52,724

311,216

Total Product SKU’s in Dataset

Image

Description

Count

Methodology

9

4/5/2024

Scoring Model

Good

Bad

Image-to-text Model (OCR & Captioning)

Text-to-Text Model

Scoring Model

Published

To Vendor

Product Data

Image Captioning Model

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding & Generation

10

4/5/2024

Source: Hugging Face - Salesforce/blip-image-captioning-base

Example Results

Image Captioning Model

More Image-to-text Models Comparison

11

4/5/2024

Source: https://huggingface.co/microsoft/git-large-coco

MS-git-large-coco

The blender is made of glass and has a glass top

The most popular cereals in the world

The first item in the package is made of black leather.

Food for the family to eat

SalesForce-BLIP2-large

Of the adidas no show sock

Of sheba meaty tuna flavor sticks

Of trix cereal

of the oster osmo pro blender

Prompt: “Generate a thorough product description”

Source: https://huggingface.co/Salesforce/blip2-opt-2.7b

The image features a pair of black socks with a blue label, placed on a white background. The socks are positioned close to each other. The socks appear to be well-maintained and clean, with no visible stains or dirt. The blue label on the socks adds a touch of color and branding to the overall appearance of the product

The image displays a box of Cocoa Puffs, a popular cereal brand, with a cartoon character on the front. The character, a cute and playful rabbit, is smiling and holding a bowl of cereal. The cereal is displayed in a bowl, with the bowl centered in the box. The box is red and white, making it stand out

UnumCloud-uform-gen

Source: unum-cloud/uform-gen · Hugging Face

OCR Model – Azure Image Analysis 4.0 API

12

4/5/2024

INPUT IMAGE

READ:

IMAGE ANALYSIS 4.0

$1 per 1000 transactions

Azure AI Vision

CAPTION:

TAGS:

General NATURALLY FLAVORED Honey.Nut Cheerios first WHOLE OATS ingredient GRAIN Har lower CHOLESTEROL® AMPLY MADE Gluten Free Sweetened Whole Grain Dat Cereal with Real Honey & Nototal Nowind Flaver NET WT 1 LB 1 OZ (17 OZ) (461g)

text, breakfast cereal, box, cereal, convenience food, food, container, indoor

a box of cereal with a red bowl of oats

Text-to-text Model

• Apply 4 pre-trained LLMs to generate product descriptions and feature summary
• Feed all available product information into LLMs
• State-of-the-art models are employed, such as GPT-3.5 and Llama-2

13

4/5/2024

Example Prompts Fed to LLMs for Product Description Generation

14

4/5/2024

Comparison of Text Generation Models

15

4/5/2024

Mixtral-8x7b

instruct

Yankee Candle Vanilla Cupcake

Gemma-7b-it

Llama-2-70b-chat

GPT-3.5

turbo

instruct

Experience the delightful aroma of freshly baked vanilla cupcakes with Yankee Candle Signature Collection Small Tumbler Vanilla Cupcake. This 4.3 Oz. candle features an irresistible fragrance with top notes of vanilla icing and malted sugar, middle notes of cake batter and chocolate, and base notes of cocoa and vanilla bean. The combination of these scents creates a warm, inviting scent that is perfect for any room in your home…

Indulge in the irresistible aroma of freshly baked vanilla cupcakes, layered with layers of buttery icing. This fragrance is reminiscent of the sweet and decadent treats enjoyed on special occasions. Top notes of vanilla icing and malted sugar create a sweet and creamy first impression. The middle notes of cake batter and chocolate bring the sweetness to the forefront, while the base notes of cocoa and vanilla bean add depth and lingered sweetness…

Are you ready to indulge your senses in the sweetest way possible? Look no further than our Yankee Candle Signature Collection Small Tumbler Vanilla Cupcake! This delightful candle combines the rich, creamy aroma of vanilla cupcakes with hints of lemon and buttery icing, creating a mouthwatering scent that's sure to satisfy your sweet tooth…

Introducing the perfect blend of sweetness and indulgence with the Yankee Candle Signature Collection Small Tumbler Vanilla Cupcake. This 4.3 oz candle has a decadent fragrance that will fill your home with the irresistible scent of vanilla cupcakes, enriched with hints of lemon and luscious buttery icing. The fragrance notes are carefully crafted to capture the essence of this classic treat, with notes of vanilla icing, malted sugar, and cake batter at the top, followed by chocolate and cocoa in the middle, and finished off with a touch of vanilla bean…

Model

Generated Product Description

ARE THE PRODUCT DESCRIPTIONS GENERATED BY LLM PRODUCTION READY?

SCORING MODEL

SCORING MODEL

17

4/5/2024

RELEVANCY

COMPLETENESS

READABILITY

CONFIDENCE SCORES

SCORING MODEL: RELEVANCY

COSINE SIMILARITY

18

4/5/2024

θ

Product Images

Product Name

Response from LLM

TEXT INPUT

TEXT OUTPUT

RELEVANCY

Cosine Similarity

Fail (<0.5)

Pass (>=0.5)

0

1

Cosine Similarity | Relevance

19

4/5/2024

Product name is {MEIJER DOG LEASH PINK LARGE} and information captured from the images are {meijer leash LARGE 08820 727961 Made In China}. The caption from a Vision AI described the image as {| a pink leash with a metal clasp | a pink leash with a white label}

Product Description: The Meijer Dog Leash Pink Large & LARGE 08820 727961 is perfect for your furry friend! This leash is a large size, and it is made with gorgeous pink material that is soft and comfortable for your dog to walk on. The leash features a strong metal clasp that securely attaches to your dog's collar, ensuring that they remain safe and secure during their walks. The white label on this particular dog leash indicates that it is an official Meijer product. If you want to keep your dog looking and feeling their best, the Meijer Dog Leash Pink Large & LARGE 08820 727961 is the perfect choice.\n\nConfidence score: 90%

Cosine similarity = 55%

vs

TEXT INPUT

TEXT OUTPUT

SCORING MODEL: COMPLETENESS

AIDA FRAMEWORK

Commonly used by marketers to understand and influence consumer behavior.

20

4/5/2024

COMPLETENESS

AIDA FRAMEWORK

ATTENTION

DESIRE

ACTION

INTEREST

SCORING MODEL: COMPLETENESS

AIDA FRAMEWORK

Commonly used by marketers to understand and influence consumer behavior.

21

4/5/2024

Prompt: On a scale of 1-10 , how likely is the Product Description generated able to capture the attention/Desire/Interest/Action of the customers?

Attention

Interest

Desire

Action

Fail (0 - 6)

Pass (6 - 10)

PRODUCT DESCRIPTIONS

SCORING MODEL: READABILITY

Flesch Kincaid Grade Level

A readability tests designed to indicate how difficult a passage in English is to understand.

22

4/5/2024

Readability

Readability = 0.39 * (words / sentences) + 11.8 * (syllables/word) - 15.59

Confidence Scores

23

4/5/2024

Prompt: On a scale of 0-100, how confident are you on the generated response?

Scoring Model

• Distinguish whether a product’s description should be further improved
• Utilize LLMs to check if a product description meets each scoring aspect
• Assign a score to each product description

24

4/5/2024

Relevancy (Cosine Similarity)

>0.5

Completeness (AIDA)

> 6

Readability (FK-Grade Level)

<10

Confidence Score

>90%

Scoring Rubric

Score

All Conditions satisfied

Any one condition - Not Satisfied

Send to Model

for improvement

Scoring Model

Publish

*Different word count thresholds are set for different product categories.

Cloud Deployment in Azure

25

4/5/2024

Product Datasets

Product Categorization & Sampling

Dependent Libraries

Scoring Model

OCR Model

(Image to Text)

26

4/5/2024

Mistral Model

(Text-to-text LLM)

Scoring Model

Storing output in csv file.

Cloud Deployment in Azure

Cloud Deployment Architecture

27

4/5/2024

Azure Machine Learning Workspace

Azure Blob Storage

Azure Blob Storage

Compute Instance:

Standard_F2s_v2

(2 cores, 4 GB RAM, 16 GB disk)

Dependent Libraries:

Azure Vision, openai

Product Descriptions.csv

INPUT

PIPELINE in DESIGNER

OUTPUT

Images.csv

Products.csv

Managed Resource

Results

28

4/5/2024

Results

29

4/5/2024

Image

Description

SCORING MODEL 2

(cosine similarity, AIDA Framework, Readability, Confidence Scores)

Failed Products with existing:

Pass

Fail

Image

Description

Products with Image only:

Results

30

4/5/2024

Product Descriptions Approved

72.5%

Business Impact

31

4/5/2024

Our project demonstrates promising results in reducing operation costs, enhancing customer satisfaction, and boosting profitability for the retailer. It also has great potential for similar applications across various industries, such as consumer goods. We hope our work facilitates further research and development in this field for broader impacts.

100% high-quality product descriptions

99% reduction original labor time & costs

+76% average quality score

​

​

WHO

HOW

IMPACT

Limitations

• Computational Cost vs. Model Sophistication: High computational demands of advanced ML models raise costs, leading to a focus on cost-effective, open-source models.​

​

• Trade-Off and Scope Limitation: Cost constraints result in compromises on model accuracy, breadth of experimentation, and the ability to test solutions across all product lines.​

​

• Project Scope was confined to products where we at least have a product image from vendors​.

​

• The threshold of our scoring model was determined based on our empirical trials, business vendors can adjust it accordingly.

​

• Due to the unavailability of data at hand, the traction in customers with good product descriptions was not quantified​.

32

4/5/2024

Recommendations & Future scope

33

4/5/2024

THANK YOU!

Dr. Matthew Lanham

34

4/5/2024

Dr. Mark Tabladillo

Business to Analytics Problem Framing

Main analytical tasks for solving missing product description problems

35

4/5/2024

Image Information Extraction: Analyze product images to identify features

Content Creation: Create a product description with identified characteristics

Evaluation: Develop a scoring framework for text quality assessment

Analytical Process

36

4/5/2024

Product Data

Methodology

37

4/5/2024

Scoring Model

Good

Bad

Image-to-text Model (OCR & Captioning)

Text-to-Text Model

Scoring Model

Published

To Vendor

Product Data

OCR Model – Azure Computer Vision API

38

4/5/2024

INPUT IMAGE

OUTPUT TEXT

COMPUTER VISION API 

$1 per 1000 transactions

Prepare Product Samples

Extract Samples from each product category to Test Models’ Performance

39

4/5/2024

Least Granular

Most Granular

1. Extract those image angles are from ‘Front’

2. Extract those image links are started with ‘http://static’

Categorize Product

Add Constraints

8 Types of Product

8 types product * 5 samples each = 

40 Total Samples

Image Captioning Model

BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding & Generation

40

4/5/2024

Source: Hugging Face - Salesforce/blip-image-captioning-base

Example Results

Image Captioning Model

More Image-to-text Models Comparison

41

4/5/2024

Source: https://huggingface.co/microsoft/git-large-coco

MS-git-large-coco

The blender is made of glass and has a glass top

The most popular cereals in the world

The first item in the package is made of black leather.

Food for the family to eat

SalesForce-BLIP2-large

Of the adidas no show sock

Of sheba meaty tuna flavor sticks

Of trix cereal

of the oster osmo pro blender

Prompt: “Generate a thorough product description”

Source: https://huggingface.co/Salesforce/blip2-opt-2.7b

The image features a pair of black socks with a blue label, placed on a white background. The socks are positioned close to each other. The socks appear to be well-maintained and clean, with no visible stains or dirt. The blue label on the socks adds a touch of color and branding to the overall appearance of the product

The image displays a box of Cocoa Puffs, a popular cereal brand, with a cartoon character on the front. The character, a cute and playful rabbit, is smiling and holding a bowl of cereal. The cereal is displayed in a bowl, with the bowl centered in the box. The box is red and white, making it stand out

UnumCloud-uform-gen

Source: unum-cloud/uform-gen · Hugging Face

Text-to-text Model

• Apply 4 pre-trained LLMs to generate product descriptions and feature summary
• Feed all available product information into LLMs
• State-of-the-art models are employed, such as GPT-3.5 and Llama-2

42

4/5/2024

Example Prompts Fed to LLMs for Product Description Generation

43

4/5/2024

Comparison of Text Generation Models

44

4/5/2024

Mixtral-8x7b

instruct

Yankee Candle Vanilla Cupcake

Gemma-7b-it

Llama-2-70b-chat

GPT-3.5

turbo

instruct

Experience the delightful aroma of freshly baked vanilla cupcakes with Yankee Candle Signature Collection Small Tumbler Vanilla Cupcake. This 4.3 Oz. candle features an irresistible fragrance with top notes of vanilla icing and malted sugar, middle notes of cake batter and chocolate, and base notes of cocoa and vanilla bean. The combination of these scents creates a warm, inviting scent that is perfect for any room in your home…

Indulge in the irresistible aroma of freshly baked vanilla cupcakes, layered with layers of buttery icing. This fragrance is reminiscent of the sweet and decadent treats enjoyed on special occasions.Top notes of vanilla icing and malted sugar create a sweet and creamy first impression. The middle notes of cake batter and chocolate bring the sweetness to the forefront, while the base notes of cocoa and vanilla bean add depth and lingered sweetness…

Are you ready to indulge your senses in the sweetest way possible? Look no further than our Yankee Candle Signature Collection Small Tumbler Vanilla Cupcake! This delightful candle combines the rich, creamy aroma of vanilla cupcakes with hints of lemon and buttery icing, creating a mouthwatering scent that's sure to satisfy your sweet tooth…

Introducing the perfect blend of sweetness and indulgence with the Yankee Candle Signature Collection Small Tumbler Vanilla Cupcake. This 4.3 oz candle has a decadent fragrance that will fill your home with the irresistible scent of vanilla cupcakes, enriched with hints of lemon and luscious buttery icing. The fragrance notes are carefully crafted to capture the essence of this classic treat, with notes of vanilla icing, malted sugar, and cake batter at the top, followed by chocolate and cocoa in the middle, and finished off with a touch of vanilla bean…

Model

Generated Product Description

Scoring Model

• Distinguish whether a product’s description should be further improved
• Utilize LLMs to check if a product description meets each scoring aspect
• Assign a score to each product description

45

4/5/2024

*Different word count thresholds are set for different product categories.

Scoring Model

• Distinguish whether a product’s description should be further improved
• Utilize LLMs to check if a product description meets each scoring aspect
• Assign a score to each product description

46

4/5/2024

Relevancy (Cosine Similarity)

>0.5

Completeness (AIDA)

> 6

Readability (FK-Grade Level)

<10

Confidence Score

>90%

Scoring Rubric

Score

All Conditions satisfied

Any one condition - Not Satisfied

Send to Model

for improvement

Scoring Model

Publish

*Different word count thresholds are set for different product categories.

Comparison of Scoring Models

47

4/5/2024

L'Oreal Paris Glow Paradise Lip Balm

Scoring Model

48

4/5/2024

• Relevancy (Cosine)
• Completeness (AIDA)
• Readability (Grade)
• confidence score

​

​

Cosine Similarity | Relevance

49

4/5/2024

Product name is {MEIJER DOG LEASH PINK LARGE} and information captured from the images are {meijer leash LARGE 08820 727961 Made In China}. The caption from a Vision AI described the image as {| a pink leash with a metal clasp | a pink leash with a white label}

Product Description: The Meijer Dog Leash Pink Large & LARGE 08820 727961 is perfect for your furry friend! This leash is a large size, and it is made with gorgeous pink material that is soft and comfortable for your dog to walk on. The leash features a strong metal clasp that securely attaches to your dog's collar, ensuring that they remain safe and secure during their walks. The white label on this particular dog leash indicates that it is an official Meijer product. If you want to keep your dog looking and feeling their best, the Meijer Dog Leash Pink Large & LARGE 08820 727961 is the perfect choice.\n\nConfidence score: 90%

Cosine similarity = 55%

vs

Completeness | AIDA Framework

50

4/5/2024

Prompt: On a scale of 1-10 , how likely is the Product Description generated able to capture the attention/Desire/Interest/Action of the customers?

(or)

Prompt: Is the Product Description generated able to capture the attention/Desire/Interest/Action of the customers? Yes/No?

​

Readability | Flesch Kincaid Grade Level

51

4/5/2024

Readability = 0.39 * (words / sentences) + 11.8 * (syllables/word) - 15.59

​

The Flesch–Kincaid readability tests are readability tests designed to indicate how difficult a passage in English is to understand.

Note: Lesser score the better

Creteria: Readability score less than or equal to 10

​

Confidence Scores

52

4/5/2024

Prompt: On a scale of 0-100 , how confident are you on the generated response?

Criteria: Accept responses only with the confidence scores of greater than 90

​

Example Prompts Fed to LLMs for Scoring

53

4/5/2024

Readability

Descriptive

Relevance

Grammar

Please check if this product description is readable and please answer yes or no: {Generated Product Description}

Please check if this product description is relevant to the product name and please answer yes or no: Product Description: {Generated Product Description} Product Name: {ProductName}

Please check if this product description uses descriptive language and please answer yes or no: {Generated Product Description}

Please check if this product description is grammatically correct and please answer yes or no: {Generated Product Description}

Comparison of Scoring Models

54

4/5/2024

L'Oreal Paris Glow Paradise Lip Balm

APPENDIX

55

4/5/2024

About Data

56

4/5/2024

ActiveUPCMarketingInformation.csv

UPCImages.csv

1

*

ERD Diagram

57

4/5/2024

Exploratory Data Analysis

ActiveUPCMarketingInformation| Shape: (311216, 6)

�

​

58

4/5/2024

Out of 309k SKUs, product descriptions are missing for 151K (49%) items

Exploratory Data Analysis

​

UPCImages | Shape: (891704, 7)

59

4/5/2024

Exploratory Data Analysis

​

ActiveUPCMarketingInformation + UPCImages

60

4/5/2024

Out of 311,216 ItemSKUs in the dataset, availability of product descriptions and the corresponding images are as follows:

Product Categories

8 product categories are included in the dataset:

• Fresh
• Gas Station
• Grocery
• Hardlines and Home
• Health and Beauty
• Pets and Consumables
• Prescriptions
• Softlines

61

4/5/2024

OCR – Azure Computer Vision API

62

4/5/2024

INPUT IMAGE

OUTPUT TEXT

COMPUTER VISION API 

$1 per 1000 transactions

Designing Scoring Model

We utilize LLMs to check if a product description meets each scoring aspect, and then our model produces a score using the variables from all aspects

• Aspects for considering
• Relevance to Product
• Clarity and Readability
• Grammatical Correctness
• Product Features Coverage
• Conciseness
• Use of descriptive Language
• Attractiveness

​

• Model: Score = f(x), x is a vector of the scoring variables
• f() function can be chosen and adjusted(i.e. weights of each variable) based on users’ preference
• x is determined manually and the values of x is assigned by LLMs

63

4/5/2024

Estimated Cost for Choosing Closed Source LLMs 

64

4/5/2024

53% Products contain Image or Text Description

• The remaining 47% products' missing description cannot be addressed by our model

65

4/5/2024

*If the MarketingDetails value of a product is null, it belongs to "No Description" category.

**If a product does not contain any images, it belongs to "No Image" category.

EDA

66

4/5/2024

3686

Tasks

Smartsheet Task Assignments

67

4/5/2024

1. Task Sheet - Smartsheet.com

Project Budget

68

4/5/2024

THANK YOU

Conclusion, call to action or contact information.

69

4/5/2024