How to manually release Python memory

In the above optimization, for every 500 users, some calculations will be performed and the results will be recorded in a disk file. I originally thought that by doing this, these results would be in the disk file and would no longer occupy memory; but in fact, the big pitfall of Python is that Python will not automatically clean up these memories. This is determined by its own implementation. There are many articles on the Internet explaining the specific reasons, so I won’t copy them here.
This article will post an experimental script of the author to illustrate that Python does have such a phenomenon of not releasing memory. In addition, a solution is also proposed, that is: del first, and then explicitly call gc.collect(). Script See below for specific effects.
Experimental environment one: Win 7, Python 2.7

from time import sleep, time 
import gc 
 
def mem(way=1): 
 print time() 
 for i in range(10000000): 
  if way == 1: 
   pass 
  else: # way 2, 3 
   del i 
    
 print time() 
 if way == 1 or way == 2: 
  pass 
 else: # way 3 
  gc.collect() 
 print time() 
   
if __name__ == "__main__": 
 print "Test way 1: just pass" 
 mem(way=1) 
 sleep(20) 
 print "Test way 2: just del" 
 mem(way=2) 
 sleep(20) 
 print "Test way 3: del, and then gc.collect()" 
 mem(way=3) 
 sleep(20)

The running results are as follows:

Test way 1: just pass 
1426688589.47 
1426688590.25 
1426688590.25 
Test way 2: just del 
1426688610.25 
1426688611.05 
1426688611.05 
Test way 3: del, and then gc.collect() 
1426688631.05 
1426688631.85 
1426688631.95

For way 1 and way 2, the results They are exactly the same. The peak memory consumption of the program is 326772KB. When sleeping for 20 seconds, the real-time memory consumption is 244820KB;
For way 3, the peak memory consumption of the program is the same as above, but the real-time memory consumption during sleep is only 6336KB.
Experimental environment two: Ubuntu 14.10, Python 2.7.3
Running results:

Test way 1: just pass 
1426689577.46 
1426689579.41 
1426689579.41 
Test way 2: just del 
1426689599.43 
1426689601.1 
1426689601.1 
Test way 3: del, and then gc.collect() 
1426689621.12 
1426689622.8 
1426689623.11

ubuntu@my_machine:~$ ps -aux | grep test_mem 
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html 
ubuntu 9122 10.0 6.0 270916 245564 pts/1 S+ 14:39 0:03 python test_mem.py 
ubuntu 9134 0.0 0.0 8104 924 pts/2 S+ 14:40 0:00 grep --color=auto test_mem 
ubuntu@my_machine:~$ ps -aux | grep test_mem 
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html 
ubuntu 9122 10.0 6.0 270916 245564 pts/1 S+ 14:39 0:03 python test_mem.py 
ubuntu 9134 0.0 0.0 8104 924 pts/2 S+ 14:40 0:00 grep --color=auto test_mem 
ubuntu@my_machine:~$ ps -aux | grep test_mem 
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html 
ubuntu 9122 11.6 0.1 30956 5608 pts/1 S+ 14:39 0:05 python test_mem.py

Conclusion:
The above description, when calling del, actually Python does not actually release the memory, but continues to put it in its memory pool; the memory is only released when gc.collect() is explicitly called.
Further:
In fact, go back to the script of the previous blog and let it introduce gc.collect(), and then write a monitoring script to monitor the memory consumption:

while ((1)); do ps -aux | sort -n -k5,6 | grep my_script; free; sleep 5; done

Result It was found that the memory will not be restored after each group of 500 users is executed, but will continue to be consumed until only about 70MB is left, and gc seems to work. In this environment, the machine uses Cloud instance, the total memory is 2G, and the available memory is about 1G. The common memory consumption of this script is 900M - 1G. In other words, for this script, gc does not take effect immediately, but when the available memory of the system drops from 1 - 1.2G to only about 70M, gc starts to take effect. This is indeed strange. I don’t know if it has something to do with the fact that the script uses gc.collect() in Thread, or if the function of gc is not controllable in the first place. The author has not done relevant experiments yet, and may continue to discuss it in the next blog.
However, it is certain that if gc.collect() is not used, the original script will exhaust the system memory and be killed. This is evident from the syslog.

The above is the detailed content of How to manually release Python memory. For more information, please follow other related articles on the PHP Chinese website!