Performance Metrics

• Data Quality
• Classification Metrics
• Regression Metrics
• Clustering Metrics
• Performance & Efficiency
• Large Language Model (LLM) & Other Model Type Specific Metrics
• Convolutional Neural Network (CNN) Specific Metrics

Business Success Metrics

• User Experience
• Operating Metrics
• Business Performance
• Suitability to Task

A linear model used for binary classification that estimates probability of a binary outcome using a logistic function. Use Case: Predicting if email is spam or not.

Classifies instances based on majority class among k-nearest neighbors in a feature space. Use Case: Recommending products based on similarities to other customers' purchase histories.

An ensemble model that constructs multiple decision trees during training and outputs the mode or mean prediction of the individual trees. Use Case: Predicting customer churn for subscriptions.

A model composed of interconnected layers of nodes (neurons) that learn to transform input data through non-linear mappings to output predictions. Use Case: Recognizing objects in images.

A model trained on vast amounts of text data to understand and generate language, capable of a variety of natural language processing tasks. Use Case: Generating human-like text for customer service chatbots or writing assistance tools.

A type of deep learning model designed to process and analyze visual data by automatically learning spatial hierarchies of features. Use Case: Detecting and classifying objects in images for applications like automated medical diagnosis from X-ray images.

We will not be going over the dozens of KPIs and metrics in depth. Instead, we’re going to focus on a top few for common use cases.

WHAT WE WON’T BE DOING

We will have a comprehensive, (though not fully exhaustive), list of metrics posted with our course materials, including: the metrics, their targets (where applicable), general notes, and references to go deeper.

WHAT YOU CAN DO

Metrics

Metrics are quantifiable measurements used to track and assess the performance of a specific process, system, or activity. They provide objective data that can be used to analyze and improve the underlying issue being measured.

Key Performance Indicators (KPIs)

This is also a measurement, but it’s related to a strategic issue. KPIs are a subset of metrics chosen to reflect the most critical or important aspects of performance for a particular goal or objective. KPIs are used to evaluate and communicate progress towards achieving strategic and operational targets and reflect how successful your business is at achieving that goal.

Suitability to Task

For all the technical assessments we need to do, in the end all that matters are the business outcomes we see. The tools are a means to that end.

Costs / P&L

In the context of business, (vs. not-for-profit or government), the only sustainable value proposition is one that is profitable. (Cost of Acquisition (CAC) vs. Lifetime Value (LTV))

Evil product thieves have been getting away with fake transactions and hurting our business. So we built a fraud detector!

For testing, we have…

​

• Total Transactions: 1000
• Actual fraudulent transactions: 100
• Actual legitimate transactions: 900

​

Model predictions

​

• Predicted fraud transactions: 130 (including true positive and false positives.)
• Predicted legitimate transactions: 870 (including true negatives and false negatives.)

​

Of the 130 transactions predicted to be fraudulent:

​

• 80 were fraudulent (true positives, TP)
• 50 were legitimate (false positives, FP)

​

Of the 870 transactions predicted to be legitimate:

​

• 850 were legitimate (true negatives, TN)
• 20 were fraudulent (false negatives, FN)

classification

TP and TN don’t necessarily sum to 100% because they only represent the correctly classified instances. They don’t include False Positives (FP) and False Negatives (FN).��Here, 93% of the total transactions were correctly classified as either fraudulent or legitimate.

The accuracy metric is calculated the same for both binary and multi-class classification. I.e., binary being something like spam vs. not spam emails, and multi-class being something like classifying types of animals, or products into categories. In our example, the 80 is “actually fraudulent” and the 850 is “actually legitimate.”

classification

Precision is on a scale from 0 to 1 or expressed as a percentage. Higher the precision is best, with 1.0 being always right in predicting the target class. (Note: for multi-class issues, precision is typically calculated separately for each class.)

Precision helps us avoid potentially high costs of having false positives. Bad predictions in our example can result in delayed or lost transactions and upset customers. We need to reduce fraud, but our model needs work if it costs us good transactions and customers. We need to reduce the false positive errors.

Be aware of the challenge… if we increase precision, we might reduce accuracy and recall.

classification

This is a 0 – 1 percentage measure. (So the higher the better here; with 1 meaning you found every event such as a fraudulent transaction in our example.)

Recall may also be known as “sensitivity” or the “true positive rate.” In our example, how many other transactions were fraud that we missed? In search, this would be how many other relevant documents were not returned. In medicine, was our test not sensitive enough to detect an illness.

Here, 80% of the actual fraudulent transactions were correctly identified.

classification

We can perhaps adjust a classification threshold.

​

We can enhance feature selection and engineering to incorporate more informative features that help distinguish between fraudulent and legitimate transactions.

​

We can try other modeling techniques / algorithms or ensemble methods to better capture fraud patterns.

​

If we do these things, we want to maintain high recall so that we continue to identify most fraudulent transactions.

We maybe have somewhat decent accuracy and recall. We’re – arguably - doing somewhat well in predicting fraud; but could we do better?

We caught a lot of fraud, but we seem to have a fairly high rate of false positives. If we took some action on these, (like ask for more customer information), we could lose sales due to customer frustration.

The goal is to increase recall (catch more fraud) while also improving precision (reduce false positives). The exact balance would depend on the specific costs associated with false positives versus false negatives in your business context.

classification

Recall

�Recall measures how many of the actual positive cases your model correctly identified. It's about completeness - how many relevant items did you catch?

​

Consider a fishing net: Recall is like casting a wide net. You want to catch all the fish (true positives), even if you catch some seaweed (false positives) too.

​

Think of a fire alarm system. Recall is like the system’s ability to detect all the actual fires. A high recall means the alarm goes off every time there is a fire, even if it sometimes goes off when there isn’t one.

Precision

�Precision measures how many of the cases your model identified as positive were actually positive. It's about accuracy - how reliable are your positive predictions?

​

Using the same fishing net metaphor, precision is like measuring how many of the things you caught in your net are actually fish (and not weeds or trash). A high precision means that most of what you caught are actual fish.�

For the fire alarm system, precision is like the alarm’s ability to avoid false alarms. A high precision means that when the alarm goes off, there is indeed a fire, and it doesn't go off for false positives like burnt toast.

Recall focuses on not missing any positive cases, while Precision focuses on ensuring that what you've identified as positive is correct. In many real-world scenarios, there's often a trade-off between the two.

classification

This set of information can offer a deeper understanding of a model’s recall, accuracy, precision, and overall effectiveness. Basic measures, (accuracy, precision, recall), may obscure underlying truths. By looking more closely at the numbers that drive those ratios, we can understand more about where our models may be doing well or not.

We did not go over the F1 score or a Precision-Recall Curve. (Those are in the fuller metrics list.) Both of those tools help view and balance precision and recall trade-off issues. The Confusion Matrix lays out the specific driver numbers behind these insightful tools.

True

Positive

False Negative�(Type II Error)

False Positive�(Type 1 Error)

True Negative

Positive

Predicted

Negative

Actual

Negative

Positive

This simple example is for a binary classification. (We have “fraud” vs. “not fraud.”

​

But we can also use the matrix for more than two classes, you just end up with a larger grid with the possibilities; e.g., different types of fraud attacks.

Large Language Models

In some ways, it’s like “Precision” in that it’s looking at how well new words fit properly into the output.

As with most of these scores, there are trade-offs… e.g., semantic similarity. If we say, “the plane is in the sky” and “the plane is flying” that should be similar, whereas “the plane is in the hanger” means something totally different. A summary that gets this wrong in this manner might not get caught by a poor BLEU score. (Because all these phrases share n-grams)

Recall that an n-gram is a sequence of n items; in our case here, words. (a bigram is 2 words, etc.) Here, we compare words from the predicted sentence with target sentences. Words matching target sentences are considered correct.

​

The simple version is computed with the fraction of candidate words in the reference. This is only the beginning though. For details, see the references within the more detailed course reference list of metrics.

We can use BLEU to test effectiveness for tasks like text summaries, translation, speech recognition, image captioning and similar.

Scoring is on 0 – 1 scale with 1 being perfect. Higher scores suggest translation – for example – is closer to the reference translation. Generally, 0-10 is poor, 50+ is good.

Large Language Models

ROUGE scores serve as an indicator of similarity based on shared words, either in the form of n-grams or word sequences. The score ranges from 0 to 1, where a higher score indicates a greater similarity. This gives insight into how well the automated summary captures relevant information.

"Gisting" refers to the process of extracting the main ideas or the essence of a text, rather than focusing on all the details.

There’s several ROUGE scores:

�ROUGE-1: Unigram (single word) overlap

ROUGE-2: Bigram (two-word sequence) overlap

ROUGE-L: Longest common subsequence

ROUGE-W: Weighted LCS (emphasizes longer sequences)

ROUGE-S: Skip-bigram (non-consecutive word pairs) overlap

We will look at an example in code soon.

Classification Metrics

Classification on News Articles with Accuracy Precision, Recall and F1 Scores

Language Model Test

Language Model Test with PENN dataset with metrics for Perplexity, Accuracy, and BLEU score.

​

Language Model Test

Language Model Test with ROUGE score.

​

Language Model Test

Language Model Test with ROUGE, BLEU and METEOR scores.

​

In many techniques, we’re trying to fit a curve to a function; something that best represents the relationship between input variables (features) and the output (target or predictions), based on observed data.

​

This is generally the case for supervised learning regression and neural networks.

​

Models like decision trees, rule-based models, clustering, ensemble, reinforcement learning and similar are not so much about curve fit. In these cases, we try to evaluate how effectively data is partitioned or optimized or makes decisions.

As in the past, the “battle of the checkboxes” of features will repeat itself in this category.

Most of these tools will have some collection of basic must have common features and then differentiate by types of metrics, automated options, and more.

We are still early in the evolution of these tools. We’ll see consolidation and and build out or acquisitions by existing analytics companies.

As with all such tools, product, dev, and finance teams will have to assess the ROI on such tools. Continuous monitoring of production products will become mission critical as these products become more embedded in core offerings.