How Can I Remove Duplicate Dictionaries from a Python List While Preserving or Ignoring Order?

Removing Duplicate Dictionaries from a List in Python

Given a list of dictionaries, the goal is to remove all dictionaries that share identical key-value pairs. For instance, transforming the list [{'a': 123}, {'b': 123}, {'a': 123}] into [{'a': 123}, {'b': 123}] would achieve this objective.

To accomplish this, you can employ a straightforward yet powerful technique:

[dict(t) for t in {tuple(d.items()) for d in l}]

This approach operates in three distinct steps:

1. Convert each dictionary into a tuple, where the tuple stores the dictionary's key-value pairs.
2. Transform the list of tuples into a set, effectively removing duplicates since sets cannot hold multiple instances of the same value.
3. Reconstruct the dictionaries from the tuples using the dict() function.

The variables in the code snippet represent the following:

• l: The original list of dictionaries
• d: An individual dictionary within the list
• t: A tuple representing that dictionary's key-value pairs

If maintaining the original order of the dictionaries is crucial, you can utilize the following modified approach:

seen = set()
new_l = []
for d in l:
    t = tuple(d.items())
    if t not in seen:
        seen.add(t)
        new_l.append(d)

This code incorporates an additional step of maintaining a seen set to track which tuples (and thus dictionaries) have already been encountered. If a tuple is not found in the seen set, it is added to the set, and the corresponding dictionary is appended to the new_l list.

It is important to note that comparing dictionaries directly using equality (==) may not always produce the desired result if the dictionaries have been modified in different ways. If the order of key-value pairs matters, it is recommended to sort the dictionary's items before creating the tuple.

The above is the detailed content of How Can I Remove Duplicate Dictionaries from a Python List While Preserving or Ignoring Order?. For more information, please follow other related articles on the PHP Chinese website!