As pointed out by a user in the comments, this question is similar in essence to another question here
Indeed, calling malloc_trim
solves my problem. I don't even have to delete the queue for it to release the memory.
import ctypes
def malloc_trim():
ctypes.CDLL('libc.so.6').malloc_trim(0)
I just added the above, and called after the tasks have completed:
await asyncio.gather(stream_coro, write_coro)
malloc_trim()
and here is the result:
Downloaded: 100%, q size: 8145
Http stream finished
write to disk finished
tasks finished... sleeping 30 seconds
mem used: 2049.73 MiB
mem used: 2049.73 MiB
mem used: 2049.73 MiB
mem used: 2049.73 MiB
mem used: 2049.73 MiB
mem used: 2049.73 MiB
Current q size: 0
running malloc_trim and sleeping 30 seconds
mem used: 33.52 MiB
mem used: 33.52 MiB
mem used: 33.52 MiB
mem used: 33.52 MiB