Home >Backend Development >Python Tutorial >How to Remove Consecutive Duplicates in Pandas?

How to Remove Consecutive Duplicates in Pandas?

Barbara Streisand
Barbara StreisandOriginal
2024-11-15 04:09:02250browse

How to Remove Consecutive Duplicates in Pandas?

Removing Consecutive Duplicates in Pandas

While Pandas' drop_duplicates() method is effective for eliminating all duplicate values, it does not discern consecutive occurrences. To address this limitation, there are efficient methods to selectively drop only consecutive duplicates.

One approach employs the shift function to compare the current value to the previous one:

a.loc[a.shift() != a]

This logic returns a mask where consecutive duplicates are characterized by False values. The loc method then selects only the rows with True values, effectively removing the consecutive duplicates.

Another method utilizes the diff function to detect changes:

a.loc[a.diff() != 0]

However, this approach is less efficient for large datasets due to the overhead associated with the differentiation calculation.

Update

It's worth noting that the default shift period is 1, so shift() and shift(1) produce equivalent results:

a.loc[a.shift(1) != a]

This ensures that the first consecutive value is correctly identified as a duplicate.

The above is the detailed content of How to Remove Consecutive Duplicates in Pandas?. For more information, please follow other related articles on the PHP Chinese website!

Statement:
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn