Data Quality

Unit – II- DWDM

Data Quality

• Data mining applications are applied to data that was collected for another purpose, or for future, but unspecified applications.
• Data mining focuses on

(1) the detection and correction of data quality problems - Data Cleaning

(2) the use of algorithms that can tolerate poor data quality.

• Measurement and Data Collection Issues
• Issues Related to Applications

​

Data Quality

​

• Measurement and Data Collection Issues
• problems due to human error,
• limitations of measuring devices, or
• flaws in the data collection process.
• Values or even entire data objects may be missing.
• Spurious or duplicate objects; i.e., multiple data objects that all correspond to a single “real” object.
• Example - there might be two different records for a person who has recently lived at two different addresses.
• Inconsistencies—
• Example - a person has a height of 2 meters, but weighs only 2 kilograms.

Data Quality

​

• Measurement and Data Collection Errors
• Measurement error - any problem resulting from the measurement process.
• Value recorded differs from the true value to some extent.
• Continuous attributes:
• Numerical difference of the measured and true value is called the error.
• Data collection error - errors such as omitting data objects or attribute values, or inappropriately including a data object.
• For example, a study of animals of a certain species might include animals of a related species that are similar in appearance to the species of interest.

Data Quality

​

• Measurement and Data Collection Errors
• Noise and Artifacts:
• Noise is the random component of a measurement error.
• It may involve the distortion of a value or the addition of spurious objects.

Data Quality

​

Data Quality

​

• Measurement and Data Collection Errors
• Noise and Artifacts:
• used in connection with data that has a spatial or temporal component.
• Techniques from signal or image processing can frequently be used to reduce noise
• These will help to discover patterns (signals) that might be “lost in the noise.”
• Note:Elimination of noise - difficult
• robust algorithms - produce acceptable results even when noise is present.

Data Quality

​

• Measurement and Data Collection Errors
• Noise and Artifacts:
• Artifacts: Deterministic distortions of the data
• Data errors may be the result of a more deterministic phenomenon, such as a streak in the same place on a set of photographs.

​

​

Data Quality

​

• Measurement and Data Collection Errors
• Precision, Bias, and Accuracy:
• Precision:
• The closeness of repeated measurements (of the same quantity) to one another.
• Precision is often measured by the standard deviation of a set of values
• Bias:
• A systematic variation of measurements from the quantity being measured.
• Bias is measured by taking the difference between the mean of the set of values and the known value of the quantity being measured.
• Example:
• standard laboratory weight with a mass of 1g and want to assess the precision and bias of our new laboratory scale.
• weigh the mass five times & values are: {1.015, 0.990, 1.013, 1.001, 0.986}.
• The mean of these values is 1.001, and hence, the bias is 0.001.
• The precision, as measured by the standard deviation, is 0.013.

Data Quality

​

• Measurement and Data Collection Errors
• Precision, Bias, and Accuracy:
• Accuracy:
• The closeness of measurements to the true value of the quantity being measured.

​

Data Quality

​

• Measurement and Data Collection Errors
• Outliers:
• Outliers are either
• (1) data objects that, in some sense, have characteristics that are different from most of the other data objects in the data set, or
• (2) values of an attribute that are unusual with respect to the typical values for that attribute.
• Alternatively - anomalous objects or values.

​

Data Quality

​

• Measurement and Data Collection Errors
• Missing Values:
• Eliminate Data Objects or Attributes
• Estimate Missing Values
• Ignore the Missing Value during Analysis
• Inconsistent Values

Data Quality

​

• Measurement and Data Collection Errors
• Duplicate Data: Same Data in multiple Data Objects
• To detect and eliminate such duplicates, two main issues must be addressed.
• First - if two objects represent a single object, then the values of corresponding attributes may differ, and these inconsistent values must be resolved
• Second - care needs to be taken to avoid accidentally combining data objects that are similar - deduplication

Data Quality “data is of high quality if it is suitable for its intended use.”

​

• Issues Related to Applications:
• Timeliness:
• If the data is out of date, then so are the models and patterns that are based on it.
• Relevance:
• The available data must contain the information necessary for the application.
• Consider the task of building a model that predicts the accident rate for drivers. If information about the age and gender of the driver is omitted, then it is likely that the model will have limited accuracy unless this information is indirectly available through other attributes.
• Knowledge about the Data:
• Data sets are accompanied documentation that describes different aspects of the data;
• the quality of this documentation can help in the subsequent analysis.
• For example,
• If the documentation is poor, however, and fails to tell us, for example, that the missing values for a particular field are indicated with a -9999, then our analysis of the data may be faulty.
• Other important characteristics are the precision of the data, the type of features (nominal, ordinal, interval, ratio), the scale of measurement (e.g., meters or feet for length), and the origin of the data.