How to Identify Rows Present in One Pandas DataFrame but Not Another?

Identifying Uncommon Rows in Pandas DataFrames

When working with multiple data frames, it becomes necessary to identify rows that exist in one but not the other. Suppose we have two data frames, df1 and df2, where df2 is a subset of df1.

How can we extract the rows from df1 that are not present in df2?

Consider the following example:

import pandas as pd

df1 = pd.DataFrame(data={'col1': [1, 2, 3, 4, 5, 3], 'col2': [10, 11, 12, 13, 14, 10]})
df2 = pd.DataFrame(data={'col1': [1, 2, 3], 'col2': [10, 11, 12]})

print("df1:")
print(df1)

print("\ndf2:")
print(df2)

Output:

   col1  col2
0     1    10
1     2    11
2     3    12
3     4    13
4     5    14
5     3    10

   col1  col2
0     1    10
1     2    11
2     3    12

Our objective is to find the rows in df1 that are not present in df2.

Solution:

To accurately identify the uncommon rows, we need to perform a left join between df1 and df2 on both col1 and col2 columns, ensuring that duplicates in df2 are eliminated. Additionally, we specify indicator=True to create an extra column indicating the source of each merged row.

The resulting data frame, df_all, contains all rows from both df1 and df2 with an additional column _merge that indicates whether a row originates from both data frames (both), only df1 (left_only), or only df2 (right_only).

df_all = df1.merge(df2.drop_duplicates(), on=['col1', 'col2'], how='left', indicator=True)

We can now filter df_all to extract the uncommon rows from df1 using the boolean condition df_all['_merge'] == 'left_only'.

df_uncommon = df_all[df_all['_merge'] == 'left_only']
print("\nUncommon rows in df1:")
print(df_uncommon)

This will return the desired output:

   col1  col2 _merge
3     4    13  left_only
4     5    14  left_only
5     3    10  left_only

By leveraging the left join with duplicate elimination and the _merge column, we can effectively identify and extract the rows from df1 that are not present in df2.

The above is the detailed content of How to Identify Rows Present in One Pandas DataFrame but Not Another?. For more information, please follow other related articles on the PHP Chinese website!