How to Efficiently Remove Duplicate Dictionaries from a Python List While Preserving Order?

Removing Duplicate Dictionaries from a List in Python

When handling a list of dictionaries, it's often necessary to remove duplicates that share identical key-value pairs. This article provides a robust solution using Python.

Problem Statement:

Given a list of dictionaries, the goal is to remove the dictionaries that contain the same key and value pairs.

Solution:

To achieve this, we employ a two-step approach:

1. Convert Dictionaries to Hashable Tuples:
   We convert each dictionary into a tuple where the elements are the key-value pairs. This step is crucial as dictionaries are not hashable, but tuples are.
2. Remove Duplicates Using a Set:
   We create a set from the list of tuples. A set automatically removes duplicates, leaving only unique tuples.

To reconstruct the dictionaries from the unique tuples, we use a dictionary comprehension. Here's the code snippet:

original_list = [{'a': 123}, {'b': 123}, {'a': 123}]

# Convert dictionaries to tuples
tuples = [tuple(d.items()) for d in original_list]

# Remove duplicates using a set
unique_tuples = set(tuples)

# Reconstruct dictionaries
result_list = [dict(t) for t in unique_tuples]

print(result_list)

Output:

[{'a': 123}, {'b': 123}]

Preserving Ordering:

If preserving the original order of the dictionaries is essential, we can use a slightly different approach:

1. Create a Seen Set:
   Initialize a set called seen to track the unique tuples.
2. Iterate over the Dictionaries:
   Iterate over the original list and for each dictionary, convert it to a tuple. If the tuple is not in the seen set, add it and append the dictionary to the result list.

Here's the code:

original_list = [{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}, {'a': 123, 'b': 1234}]

seen = set()
result_list = []

for d in original_list:
    t = tuple(d.items())
    if t not in seen:
        seen.add(t)
        result_list.append(d)

print(result_list)

Output:

[{'a': 123, 'b': 1234}, {'a': 3222, 'b': 1234}]

The above is the detailed content of How to Efficiently Remove Duplicate Dictionaries from a Python List While Preserving Order?. For more information, please follow other related articles on the PHP Chinese website!