What are the best practices for data understanding and exploration in CRISP-DM?

1 Data understanding objectives

The main objective of data understanding is to get familiar with the data sources, the data quality, the data structure, and the data distribution. This phase involves collecting, describing, exploring, and verifying the data that you will use for your data mining project. By doing this, you will be able to identify the potential opportunities and challenges for data analysis, as well as the data requirements and limitations. Data understanding is essential for defining the data mining problem, selecting the appropriate techniques, and preparing the data for modeling.

2 Data collection methods

Data collection is the first step of data understanding. It involves acquiring the data from various sources, such as databases, files, APIs, web pages, surveys, or sensors. Depending on the nature and scope of your project, you may need to collect data from multiple sources and formats, such as structured, semi-structured, or unstructured data. The best practice for data collection is to ensure that you have access to the relevant, reliable, and sufficient data that can answer your business questions. You should also document the data sources, the data formats, the data owners, and the data access methods for future reference.

3 Data description techniques

Data description is the second step of data understanding. It involves summarizing the main characteristics and features of the data, such as the number of records, the number of variables, the variable types, the variable names, the variable values, and the variable descriptions. Data description helps you to understand the scope and complexity of the data, as well as to detect any errors or inconsistencies in the data. The best practice for data description is to use descriptive statistics and metadata to provide a comprehensive overview of the data. You should also create a data dictionary that defines the meaning and format of each variable.

4 Data exploration tools

Data exploration is the third step of data understanding. It involves examining the data in more depth and detail, such as the relationships, patterns, trends, outliers, and anomalies in the data. Data exploration helps you to discover the hidden insights and information in the data, as well as to test your assumptions and hypotheses about the data. The best practice for data exploration is to use visual and analytical tools to explore the data from different perspectives and dimensions. You should also use exploratory data analysis (EDA) techniques, such as histograms, boxplots, scatterplots, correlation matrices, and principal component analysis (PCA), to reveal the distribution, variation, correlation, and dimensionality of the data.

5 Data verification methods

Data verification is the fourth and final step of data understanding. It involves validating the accuracy, completeness, and consistency of the data, as well as identifying and resolving any data quality issues, such as missing values, duplicates, outliers, errors, or noise. Data verification helps you to ensure that the data is suitable and ready for data mining, as well as to improve the performance and reliability of the data mining models. The best practice for data verification is to use data quality assessment and improvement methods, such as data profiling, data cleansing, data transformation, data integration, and data reduction.

6 Here’s what else to consider

This is a space to share examples, stories, or insights that don’t fit into any of the previous sections. What else would you like to add?