search
HomeBackend DevelopmentPython TutorialThe Python list deduplication method you should know

Preface

List deduplication is a common problem when writing Python scripts, because no matter where the source data comes from, when we convert it into a list, there may be expected The result is not our final result. The most common thing is that there are duplicate elements in the list. At this time, the first thing we need to do is deduplication.

Let’s take the simplest way first, using Python’s built-in data type set.

Suppose our list data is like this:

level_names = [
 u'Second Level',
 u'Second Level',
 u'Second Level',
 u'First Level',
 u'First Level'
]

Because the elements of the set cannot be repeated, so When converting a list into a set, duplicate elements will be automatically removed. This is the basic principle. The code is as follows:

>>> the_list = set(level_names)
>>> print(the_list)
set([u'Second Level', u'First Level'])

The disadvantage of this method is that it is converted into a list again. It is impossible to save the previous list order. If there is no such requirement, this method is the simplest answer. Maybe some friends think it is too simple. There is no technical content in this. Yes, so in general interview questions, you are asked to list. The heavy words are usually written like this:

Please write down the method to remove the heavy words from the list (set cannot be used)

People have stated that set cannot be used. , so, sometimes this trick doesn’t work, so of course it doesn’t trouble us, we have other methods.

We all know that lists can be traversed, and the problem of traversing is simple. We then define an empty list, then traverse the list with data, and add a judgment when traversing. If it is not in the empty list, add Enter it and throw it away if you have it. The code is as follows:

the_list = []
for level in level_names:
 if level not in the_list:
  the_list.append(level)
print(the_list)

Do you think this method is okay, but this method is suitable for ordinary small lists. No problem, but if you encounter a super large list, you will not be able to do what you want, because the list in the_list becomes very large, which will affect the efficiency when making judgments, because the list is searched in index order, and when the amount of data is large, it will Slow down.

Maybe you want to ask, what should I do if I encounter a large list? Is there a more awesome way? Of course, let's continue. Since using lists when judging will affect efficiency, let's change our thinking. We use sets. Then you may ask, are sets faster? Yes, because the hash function used by set When searching for values, although the set is unordered, the position is fixed. It only takes one time to check whether a specific element exists. Someone on the Internet did a comparison of searching for elements in lists and sets. Under the same data conditions, it took 16 minutes to use list. , it takes 52 seconds to use set. You can see the effect from this comparison. Not much else to say, just post the code:

the_list = []
the_set = set()
for level in level_names:
 if level not in the_set:
  the_set.add(level)
  the_list.append(level)
print(the_list)

More you For articles related to Python list deduplication methods that you should know, please pay attention to the PHP Chinese website!

Statement
The content of this article is voluntarily contributed by netizens, and the copyright belongs to the original author. This site does not assume corresponding legal responsibility. If you find any content suspected of plagiarism or infringement, please contact admin@php.cn
What are some common reasons why a Python script might not execute on Unix?What are some common reasons why a Python script might not execute on Unix?Apr 28, 2025 am 12:18 AM

The reasons why Python scripts cannot run on Unix systems include: 1) Insufficient permissions, using chmod xyour_script.py to grant execution permissions; 2) Shebang line is incorrect or missing, you should use #!/usr/bin/envpython; 3) The environment variables are not set properly, and you can print os.environ debugging; 4) Using the wrong Python version, you can specify the version on the Shebang line or the command line; 5) Dependency problems, using virtual environment to isolate dependencies; 6) Syntax errors, using python-mpy_compileyour_script.py to detect.

Give an example of a scenario where using a Python array would be more appropriate than using a list.Give an example of a scenario where using a Python array would be more appropriate than using a list.Apr 28, 2025 am 12:15 AM

Using Python arrays is more suitable for processing large amounts of numerical data than lists. 1) Arrays save more memory, 2) Arrays are faster to operate by numerical values, 3) Arrays force type consistency, 4) Arrays are compatible with C arrays, but are not as flexible and convenient as lists.

What are the performance implications of using lists versus arrays in Python?What are the performance implications of using lists versus arrays in Python?Apr 28, 2025 am 12:10 AM

Listsare Better ForeflexibilityandMixdatatatypes, Whilearraysares Superior Sumerical Computation Sand Larged Datasets.1) Unselable List Xibility, MixedDatatypes, andfrequent elementchanges.2) Usarray's sensory -sensical operations, Largedatasets, AndwhenMemoryEfficiency

How does NumPy handle memory management for large arrays?How does NumPy handle memory management for large arrays?Apr 28, 2025 am 12:07 AM

NumPymanagesmemoryforlargearraysefficientlyusingviews,copies,andmemory-mappedfiles.1)Viewsallowslicingwithoutcopying,directlymodifyingtheoriginalarray.2)Copiescanbecreatedwiththecopy()methodforpreservingdata.3)Memory-mappedfileshandlemassivedatasetsb

Which requires importing a module: lists or arrays?Which requires importing a module: lists or arrays?Apr 28, 2025 am 12:06 AM

ListsinPythondonotrequireimportingamodule,whilearraysfromthearraymoduledoneedanimport.1)Listsarebuilt-in,versatile,andcanholdmixeddatatypes.2)Arraysaremorememory-efficientfornumericdatabutlessflexible,requiringallelementstobeofthesametype.

What data types can be stored in a Python array?What data types can be stored in a Python array?Apr 27, 2025 am 12:11 AM

Pythonlistscanstoreanydatatype,arraymodulearraysstoreonetype,andNumPyarraysarefornumericalcomputations.1)Listsareversatilebutlessmemory-efficient.2)Arraymodulearraysarememory-efficientforhomogeneousdata.3)NumPyarraysareoptimizedforperformanceinscient

What happens if you try to store a value of the wrong data type in a Python array?What happens if you try to store a value of the wrong data type in a Python array?Apr 27, 2025 am 12:10 AM

WhenyouattempttostoreavalueofthewrongdatatypeinaPythonarray,you'llencounteraTypeError.Thisisduetothearraymodule'sstricttypeenforcement,whichrequiresallelementstobeofthesametypeasspecifiedbythetypecode.Forperformancereasons,arraysaremoreefficientthanl

Which is part of the Python standard library: lists or arrays?Which is part of the Python standard library: lists or arrays?Apr 27, 2025 am 12:03 AM

Pythonlistsarepartofthestandardlibrary,whilearraysarenot.Listsarebuilt-in,versatile,andusedforstoringcollections,whereasarraysareprovidedbythearraymoduleandlesscommonlyusedduetolimitedfunctionality.

See all articles

Hot AI Tools

Undresser.AI Undress

Undresser.AI Undress

AI-powered app for creating realistic nude photos

AI Clothes Remover

AI Clothes Remover

Online AI tool for removing clothes from photos.

Undress AI Tool

Undress AI Tool

Undress images for free

Clothoff.io

Clothoff.io

AI clothes remover

Video Face Swap

Video Face Swap

Swap faces in any video effortlessly with our completely free AI face swap tool!

Hot Tools

SublimeText3 English version

SublimeText3 English version

Recommended: Win version, supports code prompts!

ZendStudio 13.5.1 Mac

ZendStudio 13.5.1 Mac

Powerful PHP integrated development environment

Safe Exam Browser

Safe Exam Browser

Safe Exam Browser is a secure browser environment for taking online exams securely. This software turns any computer into a secure workstation. It controls access to any utility and prevents students from using unauthorized resources.

SublimeText3 Chinese version

SublimeText3 Chinese version

Chinese version, very easy to use

EditPlus Chinese cracked version

EditPlus Chinese cracked version

Small size, syntax highlighting, does not support code prompt function