How Can I Effectively Profile and Monitor Memory Usage in Python?

How can I analyze memory usage in Python?

Profiling Memory Usage in Python

Python 3.4 introduced the tracemalloc module for in-depth analysis of memory allocation. To display memory allocation statistics for a specific function:

from tracemalloc import start, take_snapshot, display_top

start()

# Code to profile memory usage

snapshot = take_snapshot()
display_top(snapshot)

Monitoring Memory Over Time

To track memory usage over an extended period:

from collections import Counter
import time

def count_prefixes():
    counts = Counter()
    with open('/usr/share/dict/american-english') as words:
        words = list(words)
        for word in words:
            counts[word[:3]] += 1
            time.sleep(0.0001)
    return counts.most_common(3)

count_prefixes()
snapshot = take_snapshot()
display_top(snapshot)

Using a Separate Thread for Monitoring

To monitor memory usage from a separate thread while the main thread runs:

from queue import Queue
from threading import Thread

def memory_monitor(queue):
    while True:
        try:
            command = queue.get(timeout=0.1)
            if command == 'stop':
                return
            snapshot = take_snapshot()
            print(datetime.now(), 'max RSS', getrusage(RUSAGE_SELF).ru_maxrss)
        except Empty:
            continue

queue = Queue()
monitor_thread = Thread(target=memory_monitor, args=(queue,))
monitor_thread.start()

try:
    count_prefixes()
finally:
    queue.put('stop')
    monitor_thread.join()

The above is the detailed content of How Can I Effectively Profile and Monitor Memory Usage in Python?. For more information, please follow other related articles on the PHP Chinese website!