Several methods to remove duplicates from lists in python

This article mainly introduces various methods of deduplicating lists in Python. Now I share it with you. Friends who need it can refer to it.

I encountered a problem today. After a casual prompt from a colleague, The itertools.groupby function is used. However, this thing was not used in the end.

The problem is to deduplicate the news IDs in a list, and ensure that the order remains unchanged after deduplication.

Intuitive method

The simplest idea is:

Copy code Code As follows:

ids = [1,2,3,3,4,2,3,4,5,6,1]
news_ids = []
for id in ids:
    if id not in news_ids:
        news_ids.append(id)
print news_ids

This is also possible, but it doesn’t look cool enough.

Use set

Another solution is to use set:

##Copy the code The code is as follows:

ids = [1,4,3,3,4,2,3,4,5,6,1]
ids = list(set(ids))

The result is that the original order is not maintained.

Sort again according to index

Finally solved in this way:

Copy code The code is as follows:

ids = [1,4,3,3,4,2,3,4,5,6,1]
news_ids = list(set(ids))
news_ids.sort(ids.index)

Using itertools.grouby

The article mentioned itertools.grouby at the beginning. If you don’t care about the order of the list, you can use this:

Copy code The code is as follows:

ids = [1,4,3,3,4,2,3,4,5,6,1]
ids.sort()
it = itertools.groupby(ids)
for k, g in it:
    print k

For the principle of itertools.groupby, you can see here: http://docs.python.org/2/library/itertools.html#itertools.groupby

Netizen added: Use reduce

Netizen reatlk left a message with another solution. I add and explain here:

Copy code The code is as follows:

In [5]: ids = [1,4,3,3,4,2,3,4,5,6,1]
In [6]: func = lambda x,y:x if y in x else x + [y]
In [7]: reduce(func, [[], ] + ids)
Out[7]: [1, 4, 3, 2, 5, 6]

The above is the code I run in ipython, in which the lambda x,y:x if y in x else x [y] is equivalent to lambda x,y: y in x and x or x [y] .

The idea is to first change the ids to [[], 1,4,3,...], and then use the characteristics of reduce. For an explanation of reduce, see here: http://docs.python.org/2/library/functions.html#reduce

The above is the detailed content of Several methods to remove duplicates from lists in python. For more information, please follow other related articles on the PHP Chinese website!