Monitor Python memory usage and code execution time

Which parts of my code take the longest to run and use the most memory? How can I find areas for improvement?

I’m pretty sure most of us would like to know this during development, and in this article I’ve summarized some ways to monitor the time and memory usage of your Python code.

This article will introduce 4 methods. The first 3 methods provide time information, and the fourth method can obtain memory usage.

• time module
• %%time magic command
• line_profiler
• memory_profiler

time module

This is the simplest and most straightforward (but requires manual development) method of calculating how long it takes for your code to run. His logic is also very simple: record the time before and after the code is run, and calculate the difference between the times. This can be achieved as follows:

import time
 
 start_time = time.time()
 result = 5+2
 end_time = time.time()
 
 print('Time taken = {} sec'.format(end_time - start_time))

The following example shows the difference in time between a for loop and a list comprehension:

import time
 
 # for loop vs. list comp
 list_comp_start_time = time.time()
 result = [i for i in range(0,1000000)]
 list_comp_end_time = time.time()
 print('Time taken for list comp = {} sec'.format(list_comp_end_time - list_comp_start_time))
 
 result=[]
 for_loop_start_time = time.time()
 for i in range(0,1000000):
 result.append(i)
 for_loop_end_time = time.time()
 print('Time taken for for-loop = {} sec'.format(for_loop_end_time - for_loop_start_time))
 
 list_comp_time = list_comp_end_time - list_comp_start_time
 for_loop_time = for_loop_end_time - for_loop_start_time
 print('Difference = {} %'.format((for_loop_time - list_comp_time)/list_comp_time * 100))

We all know that for will be slower.

Time taken for list comp = 0.05843973159790039 sec
 Time taken for for-loop = 0.06774497032165527 sec
 Difference = 15.922795107582594 %

%%time Magic command

Magic command is a convenient command built into the IPython kernel that can easily perform specific tasks. Generally, it is used in jupyter notebook.

Add %%time at the beginning of the cell. After the cell execution is completed, the time spent on the cell execution will be output.

%%time
 def convert_cms(cm, unit='m'):
 '''
Function to convert cm to m or feet
'''
 if unit == 'm':
 return cm/100
 return cm/30.48
 
 convert_cms(1000)

The results are as follows:

CPU times: user 24 µs, sys: 1 µs, total: 25 µs
 Wall time: 28.1 µs
 
 Out[8]: 10.0

The CPU times here are the actual time spent by the CPU processing the code, and the Wall time is the real time that the event has passed. At the method entrance and The time between method exits.

line_profiler

The first two methods only provide the total time required to execute the method. Through the time analyzer we can get the running time of each code in the function.

Here we need to use the line_profiler package. Use pip install line_profiler.

import line_profiler
 
 def convert_cms(cm, unit='m'):
 '''
Function to convert cm to m or feet
'''
 if unit == 'm':
 return cm/100
 return cm/30.48
 
 # Load the profiler
 %load_ext line_profiler
 
 # Use the profiler's magic to call the method
 %lprun -f convert_cms convert_cms(1000, 'f')

The output results are as follows:

Timer unit: 1e-06 s
 
 Total time: 4e-06 s
 File: /var/folders/y_/ff7_m0c146ddrr_mctd4vpkh0000gn/T/ipykernel_22452/382784489.py
 Function: convert_cms at line 1
 
 Line # Hits Time Per Hit % Time Line Contents
 ==============================================================
1 def convert_cms(cm, unit='m'):
2 '''
3 Function to convert cm to m or feet
4 '''
5 1 2.0 2.0 50.0 if unit == 'm':
6 return cm/100
7 1 2.0 2.0 50.0 return cm/30.48

You can see that line_profiler provides detailed information about the time spent on each line of code.

• Line Contents: The code being run
• Hits: The number of times the line was executed
• Time: The total time spent (i.e. the number of hits x the number of hits per time )
• Per Hit: The time it takes for an execution, that is, Time = Hits X Per Hit
• % Time: The proportion of the total time

You can see Yes, each line of code analyzes the time in detail, which is quite helpful for us to analyze the time.

memory_profiler

Similar to line_profiler, memory_profiler provides line-by-line memory usage of the code.

To install it you need to use pip install memory_profiler. Here we monitor the memory usage of the convert_cms_f function.

from conversions import convert_cms_f
 import memory_profiler
 
 %load_ext memory_profiler
 
 %mprun -f convert_cms_f convert_cms_f(1000, 'f')

The convert_cms_f function is defined in a separate file and then imported. The results are as follows:

Line # Mem usage Increment Occurrences Line Contents
 =============================================================
1 63.7 MiB 63.7 MiB 1 def convert_cms_f(cm, unit='m'):
2 '''
3 Function to convert cm to m or feet
4 '''
5 63.7 MiB 0.0 MiB 1 if unit == 'm':
6 return cm/100
7 63.7 MiB 0.0 MiB 1 return cm/30.48

memory_profiler provides detailed insight into the memory usage of each line of code.

1 MiB (MebiByte) here is almost equal to 1MB. 1 MiB = 1.048576 1MB

But memory_profiler also has some disadvantages: it queries the operating system memory, so the results may be slightly different from the python interpreter. If you run %mprun multiple times in a session, you may notice an increase. The measurement column reports 0.0 MiB for all code lines. This is due to the limitations of magic commands.

Although memory_profiler has some problems, it allows us to clearly understand memory usage and is a very useful tool for development.

Summary

Although Python is not a language known for its execution efficiency, these commands are still very helpful to us in some special cases.

The above is the detailed content of Monitor Python memory usage and code execution time. For more information, please follow other related articles on the PHP Chinese website!