Machine Learning Hackathon - ExxonMobil

Solution to Problem Statement 1

Team Members:

• Divakar Verma (+91-7888730471, f2014093@pilani.bits-pilani.ac.in)
• Akanksha Dara (+91-9828618771, f2014104@pilani.bits-pilani.ac.in)
• Sharat Chandra (+91-8740912916, f2014108@pilani.bits-pilani.ac.in)

Problem Statement

Currently, ExxonMobil sponsors campaign signage for retailers and field staff has to physically visit the retail sites to validate the signage to ensure the signage is within ExxonMobil brand guidelines. This manual effort will be very time consuming. Develop a solution which automates some of this work:

a.) Retailers take an image of the campaign signage and send/upload to the technology platform.

b.) The technology will leverage a combination of image analytics and machine learning to automate the validation process with the uploaded campaign signage image and identify those signage which does not confirm to ExxonMobil brand guidelines.

​

​

Solution Approach

• We design, develop and implement a modular system which can identify images with rule-compliant signage, provide the ability to train the underlying Machine Learning model, perform logo detection and geotagging -- all via a simple web portal.
• We perform the tasks using:
• Convolutional Neural Network: state of the art Deep Learning model for image classification.
• Advanced Image Analysis Techniques: Template Matching using HAAR Classifiers for accurate Logo detection.
• Geotagging: Extraction of geo-coordinates from the image.

Model Architecture

​

​

​

Image Input

Neural Network Module

Geotagging Module

Signage Detection

Learning Module

Logo Object Detection

Geographic Info

Image Analysis Module

Deep Learning: Compliant Brand Signage Identification

• We used transfer learning on Inception-v3 (Google’s cnn pre-trained on thousands of images) to train our neural network to classify images into two categories:
• Correct Brand Signage
• Incorrect Brand Signage
• After training, we tuned the parameters of the Neural Network to optimize the results for the specific use case.

​

​

​

​

Architecture of Convolutional Neural Networks converting input image to probabilistic classification outputs

Signage Detection using Neural Network Model

0.92 Correct

Input Image

Neural Network

Output

0.08 Incorrect

Image Analysis: Logo Detection

• We leveraged OpenCV2 (a library of advanced real-time computer vision algorithms) to perform image analysis.
• We implemented a scaled-variant version of template matching algorithm to identify and locate the Exxon Mobil logos in the uploaded images.
• We first created multiple templates using the logos available online and the template matching algorithm was applied at various scales thus making our logo detection size-invariant.

Demonstration

The video shows the working of the algorithm on 2 images.

• An image is resized to various scales (enlarged and shrunk).
• ExxonMobil logo is then searched on all the resized versions of the image.
• The best version is picked and is outlined.

Logo Detection Results

Step 1: Select the image to be classified and upload it to the server.

​

​

​

​

​

​

Step 2: The image is received by the backend and fed into our system consisting of Machine Learning Model, Image Analysis Engine and Geotagging module.

Step 3: The modules produce respective outputs on the backend which are then delivered to the front end.

Detected

Logo

Neural Network Output

Geotagging

• We extracted the latitude and longitude information from the image metadata.
• Used Google Maps API to:
• Get the street address of the location pointed out by the (latitude, longitude) pair.
• Point that location on the map.
• Thus, the system has an ability to identify and authenticate the geo-location of where the image was taken.
• A sample output on next slide...

Training

• We created an interface that enables Exxon Mobil to retrain the platform according to their evolving requirements just by click of a button on our portal.
• Training the platform requires the user to upload two sets of images:
• Positive Images: Represent correct brand signage
• Negative Images: Represent incorrect brand signage
• Future work may involve creating a feedback loop by human intervention and correcting the model’s mistakes in an online fashion.

Platform Training Methodologies

Model

Positive Training Set

Negative Training Set

OFFLINE TRAINING

ONLINE TRAINING

Prediction

Feedback

The signage classifier can be retrained by uploading images from both the categories to tune the model as per business requirements.

Thank You!

--Link to code--