How Do You Validate Your Data is Clean?
如何验证您的数据是否“干净”？
The data cleansing process entails several steps to identify and fix problem entries. The first step is to analyze the data to identify errors. This may involve using qualitative analysis tools that use rules, patterns, and constraints to identify invalid values. The next step is to remove or correct errors.
Common data cleaning steps include remediating:
Duplicate data: Drop duplicate information
Irrelevant data: Identify critical fields for the particular analysis and drop irrelevant data from the analysis
Outliers: Outliers can dramatically affect model performance, so identify outliers and determine appropriate action
Missing data: Flag and drop or impute missing data
Structural errors: Correct typographical errors and other inconsistencies, and make data conform to a common pattern or convention
数据清洗过程包含几个必需的步骤，旨在识别和修复问题条目。第一步是分析数据以识别错误。这可能涉及通过使用规则、模式和约束来识别无效值的定性分析工具。下一步是删除或更正错误。
常见的数据清洗步骤包括修复：
重复数据：删除重复信息
不相关数据：确定特定分析的关键字段并从分析中删除不相关数据
异常值：异常值会显著影响模型性能，因此请识别异常值并确定适当的操作
缺失数据：标记并删除或估算缺失数据
结构错误：纠正印刷错误和其他不一致，并使数据符合通用模式或约定