Why Can\'t I Allocate Large NumPy Arrays on Ubuntu Despite Sufficient Memory?

Memory Allocation Issues with Numpy Array Allocation on Ubuntu

When allocating significant memory to Numpy arrays on Ubuntu 18, users may encounter an error like "Unable to allocate array with shape and data type." This issue arises despite having sufficient memory, and it's not observed on MacOS.

Root Cause

The issue stems from the default overcommit mode (0) on Ubuntu. In this mode, the kernel restricts memory allocation to prevent overcommitting physical pages. This becomes a concern when assigning memory for sizeable arrays like the one specified, which requires approximately 282 GB.

Solution

To allocate the array successfully, you can change the overcommit mode to 1. This effectively enables "always overcommit" mode, allowing the allocation regardless of whether the physical memory is sufficient.

To set the overcommit mode to 1 as root, execute the following command:

$ echo 1 > /proc/sys/vm/overcommit_memory

Sparse Array Considerations

This solution allows you to allocate space for arrays that are sparse or mostly empty. However, it's important to note that you should write to the array explicitly to allocate physical pages where data exists. This can be crucial for efficient memory utilization.

The above is the detailed content of Why Can\'t I Allocate Large NumPy Arrays on Ubuntu Despite Sufficient Memory?. For more information, please follow other related articles on the PHP Chinese website!