How to clean data in data analysis?

In data analysis, we focus on data, but not every data is what we need to analyze, which requires us to clean the data, through the cleaning of the data, so that we can ensure that the data analysis produces a good result, so a clean data can improve the efficiency of data analysis, therefore, data cleaning is a very important work, through the cleaning of data, we can unify the format of data, so as to reduce many problems in data analysis, This improves the efficiency of data analysis. But what kind of data does it need to be cleaned? Generally speaking, the objects of data cleansing are missing values, duplicate values, outliers, etc.

First of all, let me explain to you what is a duplicate value, the so-called duplicate value, as the name suggests, is duplicate data, the same data in the data is duplicate data, duplicate data generally has two situations, the first is multiple data records with exactly the same data value. The other is when the data subjects are the same but the unique attribute values are different. One of these two situations is duplication.

So how do you deduplicate data? Generally speaking, there are only two ways to deal with duplicate data, deduplication is the solution in the first case, and removal is the solution in the second case.

Secondly, let me tell you what an outlier is, and the outlier here refers to the measured value in a set of test values where the deviation of the average of the universe exceeds two standard deviations. A measured value that deviates more than three standard deviations from the mean is called a height outlier. We generally don't deal with outliers, but of course, the premise is that the algorithm is not sensitive enough to outliers.

If the algorithm is sensitive to outliers, how to deal with outliers? Then we need to replace it with an average value, or treat it as an outlier, which can reduce the occurrence of outliers in the data.

The missing value is also the object that needs to be cleaned up in data analysis, the so-called missing value is the grouping and missing of data due to the lack of information in the data, which is called the missing value, and the data with missing value is incomplete because one or some data are not complete, which has a certain impact on data analysis. So, we need to clean up the missing values, so how do we clean up the missing values? For missing values with large samples, we can delete them directly, but if the samples are small, we can't delete them directly, because small samples may affect the final analysis results.

For small samples, we can only clean up by estimates.

The data that needs to be clear about data analysis is the duplicate values, outliers and missing values introduced in this article, and we must pay attention to these useless data when cleaning the data, only in this way can we do a good job in data analysis. Finally, we remind everyone that you must save your original data before cleaning the data, so that we can make a good backup of the data. Remember, remember.

Data cleaning is a very important step in the data analysis process, the purpose of which is to ensure the accuracy, completeness and consistency of the data, so that the subsequent data analysis work can get the correct results. In order to ensure the accuracy of data cleansing results, the following aspects can be carried out:

Set data cleaning rules: Before data cleaning, you need to formulate corresponding data cleaning rules according to the requirements and characteristics of data analysis, including missing value processing, outlier value processing, and duplicate value processing, to ensure that the data cleaning work meets the unified standards.

Carefully review the results of the data cleansing: The results of the data cleansing need to be carefully reviewed for omissions or errors, and the results of the cleansing should be verified to ensure the correctness of the data.

Leverage multiple data analysis methods: The accuracy of the data cleansing results can be further validated by using a variety of different data analysis methods to analyze the data.

Establish data cleansing logs: Establish data cleansing logs to record all data cleansing processes, including data sources, data cleansing rules, data cleansing results, and raw data, which can help troubleshoot data anomalies and trace the root cause of data problems.

Professional technical support: If conditions permit, you can seek professional data cleaning institutions or technical support to ensure the accuracy and reliability of data cleaning work.

In summary, in order to ensure the accuracy of data cleansing results, it is necessary to establish standardized data cleansing rules, conduct careful review, use multiple analysis methods, establish data cleansing logs, and seek professional technical support. Only in this way can we truly ensure the effect of data cleansing, so as to obtain correct data analysis results.

There are industry experts in this area, and we can go to them to solve this problem, and we use unnamed penguins.

Data cleansing, also known as data cleansing, is used to detect and correct (or delete) inaccurate or corrupted records in a recordset, table, or database. Broadly speaking, data purging or purging refers to identifying incorrect, incomplete, irlevant, inaccurate, or otherwise problematic parts of data and then replacing, modifying, or deleting that dirty data.

Significance of data cleansing: In simple terms, data cleansing is often considered to be a useless part (incomplete data that does not affect the result). But it's a valuable process that can help businesses save time and increase efficiency.

Data cleansing is the final process of discovering and correcting identifiable errors in data files, including checking data consistency, handling invalid and missing values, and so on. Unlike questionnaire review, post-entry data cleanup is generally done by a computer rather than a human being.

Data cleansing, as the name suggests, refers to the final process of finding and correcting recognizable errors in data files, including checking data consistency, dealing with invalid and missing values, etc. Because the data in the data warehouse is a collection of subject-oriented data, which is extracted from multiple business systems and contains historical data, it is inevitable that some data is wrong data and some data are in conflict with each other, and these wrong or conflicting data are obviously what we don't want, which is called "dirty data". We need to "wash out" the "dirty data" according to certain rules, which is data cleaning.

The task of data cleaning is to filter the data that does not meet the requirements, and submit the filtering results to the business department to confirm whether it is filtered out or corrected by the business unit before extraction. There are three main categories of data that do not meet the requirements: incomplete data, wrong data, and duplicate data. Data cleaning is different from questionnaire review, and data cleaning after entry is generally done by a computer rather than a manual.

Content from user: Bao Xi Ge.

Data preprocessingData cleaning is the process of removing errors and inconsistent data, of course, data cleaning is not simply recorded with updated data, in the process of data mining, data cleaning is the first step, that is, the process of preprocessing data. The task of data cleansing is to filter or modify data that does not meet the requirements. There are three main categories of data that do not meet the requirements: incomplete data, wrong data, and duplicate data.

A variety of different mining systems are designed for data cleaning for specific application areas. Includes:

1) Detect and eliminate data anomalies.

2) Detect and eliminate approximate duplicate records.

3) Integration of data.

4) Domain-specific data cleansing.

The data in the project is in a data warehouse, where the data is incomplete, noisy, and inconsistent. The data cleansing process attempts to fill in missing values, smooth out noise, and identify outliers, and correct inconsistencies in the data. The purpose of data cleaning is to provide accurate and effective data for mining and improve mining efficiency.

The following describes the process of data cleansing, which follows the processing process of the cloud platform.

There are two things for the data in the dataset:

1) If there are a large number of missing values in the data, we usually take the measure to delete them directly, but in some systems, when ETL processing, a large number of missing values cannot be directly processed.

2) For the more important attributes, there will also be a small number of missing values, and a series of data mining needs to be carried out after the data is completed.

In view of these two incomplete data characteristics, the following two methods are used to fill in the data during data cleaning:

1) Manually select the missing attribute values with the same constant attribute.

Remove duplicates.

Null value padding. Unified unit.

Whether the treatment is standardized.

Remove unnecessary variables.

Whether the logical value is bug-checked.

Whether new calculated variables need to be introduced.

Whether or not you need to sort.

Whether principal component or factor analysis is performed.

Wait, and many more.

Data cleansing is a step in which data needs to be preprocessed after data is entered, and only properly processed data can be entered into data mining. Whereas, processing data includes the processing of the quantity and quality of data.

Including adding or deleting the missing data related to the method, the specific steps are judged by yourself, if the amount of data is very small and still insist on deleting, it is your own problem.

Supplement: Lagrangian interpolation or Newtonian interpolation is commonly used, which is also quite easy to understand, and belongs to the basic knowledge of mathematics and physics. (The pandas library comes with a Lagrangian interpolation function, and this advantage is that the data can also be detected for outliers before interpolation, and if it is anomalous, then the data is also considered to be an object that needs to be interpolated.)

Deletion: This is easy to understand, that is, the deletion of data that has no direct impact on the analysis of the results.

Whether or not outliers are eliminated depends on the situation.

Re-interpolation as in question 1 as missing values.

Delete records that contain outliers (which may result in insufficient sample size and alter the original distribution).

Mean correction (with the average of the two observations before and after).

To sum up, the plan is still reliable.

Life is short, learn python well

3 There is too much data, and there are three methods: integration, specification, and transformation.

1) When the data is scattered, it means that the data is extracted from multiple scattered data warehouses, which may cause redundancy. What needs to be done at this time is [Data Integration].

There are two aspects to data integration:

Redundant Attribute Identification Contradictory Entity Recognition.

Attributes: For redundant attributes, my personal understanding is that the attributes with relevance are called out from different warehouses and integrated into the new table, and the new table is redundant due to too many attributes, so you can rely on correlation analysis to analyze the correlation coefficient between attribute A and attribute B to measure the extent to which one attribute contains another attribute. Wait a minute.

There are two main things that the preprocessing phase does when data is cleaned:

One is to import the data into the processing tool. Generally speaking, it is recommended to use a database to build a MySQL environment with a single number of runs. If the amount of data is large (more than 10 million), you can use text file storage + python operation.

The second is to look at the data. There are two parts here: one is to look at the metadata, including field explanations, data **, **tables, and all other information that describes the data; The second is to extract a part of the data, use manual viewing, have an intuitive understanding of the data itself, and initially find some problems to prepare for subsequent processing.

Data cleaning is an indispensable part of the whole data analysis process, and the quality of the results is directly related to the model effect and final conclusion. In practice, data cleaning usually takes up 50%-80% of the analysis process.

There are three methods for cleaning data, namely binning method, clustering method, and regression method.

1. Dividing the box.

It is a method that is often used, the so-called binning method, which is to put the data that needs to be processed into the box according to certain rules, and then test the data in each box, and take the method to process the data according to the actual situation of each box in the data.

2. Regression method.

The regression method uses the data of the function to draw the image, and then smooths the image. There are two types of regression methods, one is unilinear regression and the other is multilinear regression. Unilinear regression is all about finding the best straight line for two attributes, and being able to get the best line from one property to the other.

Multilinear regression is all about finding many properties and fitting the data to a multidimensional surface, so that the noise can be removed.

3. Clustering method.

The workflow of the clustering method is relatively simple, but the operation is really complicated, the so-called clustering method is to group abstract objects into a set of different sets, and find unexpected solitary points in the set, which are noise. This makes it possible to spot the noise directly and then remove it.