Got the similar issue here. "numpy._core._exceptions._ArrayMemoryError: Unable to allocate 3.58 GiB for an array with shape 500000000"

My PC has 16G mem, but it is not able to allocate 3.58G for an array. It seems to be the contiguous memory management issue "NumPy requires a contiguous block of memory to allocate the array. Even if your system has enough total memory, it might not have a single contiguous block large enough to satisfy the request".

Maybe try using mem-mapped array in numpy: data = np.memmap('data.dat', dtype='int64', mode='w+', shape=(500000000,)) Or breaking the dataset into smaller chunks and process them separately.