Using an extremely long List in Python results in excessive memory usage

Question

I need to use a List with a length of more than 500,000 in a program, but the result is that the memory usage increases instantly

What method should be used for optimization?

Answer 1

Does this list have to be stored in memory?

If you are processing them one by one and do not need to store them all in memory, you can write the contents to a disk file first, then open it and use a statement similar to the following:

for item in file('xxx.txt').xreadlines():
    ... # 解析出你的内容，并处理