Home >Backend Development >Python Tutorial >Python deduplicates multi-attribute duplicate data
The following is an example of Python deduplication of multi-attribute duplicate data. It has a good reference value and I hope it will be helpful to everyone. Let’s take a look together
Steps to deduplicate data in the pandas module in python:
1) Use the duplicated method in DataFrame Returns a Boolean Series to display whether there are duplicate rows in each row. No duplicate rows are displayed as FALSE, and duplicate rows are displayed as TRUE;
2) Use the drop_duplicates method in the DataFrame to return a removed DataFrame with repeated rows.
Note:
If no parameters are set in the duplicated method and drop_duplicates method, these two methods will judge all by default. If in These two methods add specified attribute names (or column names), for example: frame.drop_duplicates(['state']), then specify some columns (state columns) to determine duplicates.
Specific examples are as follows:
>>> import pandas as pd >>> data={'state':[1,1,2,2],'pop':['a','b','c','d']} >>> frame=pd.DataFrame(data) >>> frame pop state 0 a 1 1 b 1 2 c 2 3 d 2 >>> IsDuplicated=frame.duplicated() >>> print IsDuplicated 0 False 1 False 2 False 3 False dtype: bool >>> frame=frame.drop_duplicates(['state']) >>> frame pop state 0 a 1 2 c 2 >>> IsDuplicated=frame.duplicated(['state']) >>> print IsDuplicated 0 False 2 False dtype: bool >>>
##
The above is the detailed content of Python deduplicates multi-attribute duplicate data. For more information, please follow other related articles on the PHP Chinese website!