What are the key steps in preparing your data for machine learning?

Structure is the fundamental aspect of machine learning. As data structure is important in classification, your ml exercise depends on the quality and structure of your data. Additionally, understanding what data is available as well as how complete the data is is paramount at the onset of your assignment.

Data Collection: Gather the data from various sources. Data Exploration: Perform an initial exploration to understand the data's structure, distribution, and nature. This involves summarizing statistics, visualizing data distributions, and identifying potential patterns or anomalies. Data Cleaning: Includes handling missing values by identifying and addressing missing data through methods like imputation (mean, median, mode) or removal of incomplete records if appropriate. Outlier Detection: Detect and handle outliers, which can be achieved using statistical methods or visualization techniques. Data Consistency: Ensure consistency in data entry, such as uniform units of measurement, consistent naming conventions, and standard formats.

Apesar de parecer simples em alguns casos, o tratamento das bases de dados parece ser fundamental para implementar o aprendizado de máquina. No contexto de imputação de valores ausentes, pesquisas recentes que participei apresentaram resultados super interessantes a respeito do impacto gerado por bases de dados tratadas com técnicas de imputação (de simples a complexas), quando comparadas com bases de dados que não passaram pelo processo de imputação. Com certeza o tema não deve ser ignorado!

To optimally train machine learning models, it is crucial to clean the training data through the following practical steps: - Remove columns with a single value - Consider columns with very few unique values - Remove duplicate rows - Handle missing values - Identify and treat outliers - Fix structural errors - Validate data types - Address data imbalances - Normalize or standardize numerical features

Imagine you have a lot of information, but not all of it is useful. By picking out only the key pieces, you can make your model work better and faster. There are different ways to do this: some methods look at how important each piece of information is on its own, others test combinations of information to see which ones work best together, and some methods automatically choose the important parts while the model is being built. By focusing only on the essential information, your model becomes more accurate and efficient.

This step involves normalizing or standardizing numerical data so that it has a consistent scale, which helps the model learn more effectively. It also includes encoding categorical data into numerical values using methods like one-hot encoding or label encoding, making it understandable for algorithms. Additionally, feature engineering creates new useful features from the existing data, capturing more insights for the model. Data transformation ensures that the input data is clean, consistent, and enriched, ultimately improving the model's performance and accuracy.

A common rule of thumb is to use a 60/20/20 split or 70/20/10 split, where the largest portion is for training, followed by validation and test sets. Some key considerations for splitting data: - Larger training sets tend to reduce overfitting risk, but very large validation/test sets may be needed for highly complex models. - Data should be split in a stratified manner to maintain class proportions across sets for classification problems. - If data is limited, techniques like cross-validation can be used instead of a dedicated validation set.

Data readiness is critical for AI/ML success. Identifying business critical data is a key step. Get SMEs and process experts to be data owners (establish data governance) helping specify data-types and sources, including assuring data quality for accuracy. Also, easy access to data from various databases, especially when dealing with global assets. Ensure IT understands and supports accessibility without compromising security and DQ metrics. Data bias, lineage and compatibility with ML algorithms are other factors to consider. I would highly recommend using data professionals (experts in technical data) from the onset to guide and be an integral part of IT, business and DS teams for ML/AI success.