2025:

same problem here.

my findings:

with pyarrow.set_memory_pool(pyarrow.jemalloc_memory_pool()) and pyarrow.jemalloc_set_decay_ms(0) the memory will be released eventually if you have enough memory to prevent OOM kill before the full gc triggered.