I guess you are running is on raspberry pi 4 with 2gb right now? I did the same like you before and turn out that raspberry pi can not handle "heavy code" and i think some of the package inside virtual environment is new which use more ram. I hope it help.

Maybe write a C++ version could potentially save more ram usage and more efficient code