DS161 Introduction to Data Science & Artificial Intelligence

Lecture 8

Data Quality, Cleaning and Integration

Krishnendu Ghosh

What is Data Integration?

What is Data Integration?

Data Integration

Data Integration

How Data Integration Works?

Data Integration

Data Integration: Data Lakes

Data Lakes vs Data Warehouses

Importance of Data Integration

Types of Data Integration

Types of Data Integration

Types of Data Integration

Types of Data Integration

Data Integration ⇒

Data Quality Issues

Data Integration Tools

Data Integration Challenges

Data Integration Challenges

Data Integration Challenges

Data Integration: Summary

Data Quality Dimensions

Data Quality

• Completeness
• This attribute simply asks whether the data has all the vital characteristics to be considered usable and to solve the problem at hand.

Example: If you want to map your customer locations, you need to have the customer addresses. Any data point that misses this information is incomplete.

• When collecting data, in this case, you must ensure that all the vital information needed for this specific purpose is captured.
• Data is only complete if it includes bits of this essential information that would help answer the impending questions you have at hand.

Data Quality

Accuracy

• Accuracy in data relates to how the data represents the real world.

​

Example: When collecting data about our customers, you must ensure that these details are correct and accurate to the particular customers when you collect phone numbers and date of birth. When you realize that the phone numbers are wrong, they cannot be used to contact your customers, which means they are inaccurate.

Data Quality

Validity

• When you check your data and realize there are customers over 100
• years old and others less than 15 years old, your data is more likely to be invalid.
• However, this also depends on your business activities.
• If you are collecting data about the end of care homes, ages above 90 and 100 are likely in some cases.
• If your data is about children’s games, getting people over 20, 30, and 50 players is expected to be invalid data.

Data Quality

Consistency

• It mainly relates to data accuracy.
• Data consistency checks whether the data stored in one location matches similar data stored in another.
• If you have customer data in your sales data records and similar records in your accounts department, consistency dimensions check whether the data is similar and to what percentage.
• Suppose the data is different because some errors or discrepancies in one record are supposed to match another. In that case, the data quality is suspicious and needs to be monitored and corrected.

Data Quality

Integrity

• It is about maintaining the data’s true state across its lifecycle.
• When the data loses its integrity, the related data record becomes
• invalid, inaccurate, or inconsistent, which affects its overall quality.
• In other instances, data integrity relates to the safety of the data and its compliance with different data regulatory policies such as GDPR.

Data Quality

Uniqueness

• The data record should appear once in the dataset.
• Uniqueness ensures no duplicate values in your dataset. It reduces cases of overlaps.
• Having a single entry for each data point makes it possible to ensure compliance and customer engagement.

Code