Why Does NumPy Fail to Allocate Large Arrays on Ubuntu, and How Can I Fix It?

Unable to Allocate Array with Shape and Data Type

Symptom

When allocating large arrays in NumPy on Ubuntu 18, users may encounter a "MemoryError: Unable to allocate array with shape" error. This occurs despite sufficient system memory. Notably, no such issue arises on macOS.

Cause

The issue stems from the system's overcommit handling mode. In the default mode (0), the kernel may reject allocation requests deemed too excessive for physical memory.

Resolution

To resolve the error:

1. Check the overcommit mode:

   $ cat /proc/sys/vm/overcommit_memory

   If it returns 0, proceed to step 2.

2. Enable "always overcommit" mode as root:

   $ echo 1 > /proc/sys/vm/overcommit_memory

3. Retry the array allocation.

Example

For an array of dimensions (156816, 36, 53806), with a uint8 data type:

import numpy as np

# Allocate array with "always overcommit" mode enabled
a = np.zeros((156816, 36, 53806), dtype='uint8')
print(a.nbytes)  # 303755101056 bytes

The system will allocate memory only when specific array elements are written to. This allows for the use of sparse arrays.

The above is the detailed content of Why Does NumPy Fail to Allocate Large Arrays on Ubuntu, and How Can I Fix It?. For more information, please follow other related articles on the PHP Chinese website!