Data Cleaning

Cleaning Data Correctly

• Cleaning of data prior to drawing conclusions is not only a good idea, but often mandatory for your models to be able to make sense of the data

​

• Removing extraneous information can improve your modelling, but what if that data was not as useless as you thought? Data cleaning is a double-edged sword.

​

• Your conclusions may change depending on what you kept, what you removed, and how you cleaned/filtered your information up front. Know that this affects your results.

Removing data loses information and can potentially waste more data along the way, especially with larger and larger data sets

By removing noise and unnecessary “Dirty Data” from your information, it may be easier to build better models. Remember that all the data removed = information lost.

Food For Thought

• Think about what distinguishes “clean” data from “dirty” specifically. What sort of signs can you look for to tell if a set needs cleaning? How does the content of the data and what it represents affect what you classify as useful or not useful?

​

• Come up with an example scenario where the act of “cleaning” data has negative consequences for either your conclusions or the people who are affected by your work. Think about what sorts of data, containing what information, who might be harmed by removal or reshaping of certain subsets or pieces of data.