Python worker is getting killed mid stream when the data is serialized and sent via socket to the python worker. For some reason, the python worker crashes unexpectedly and the socket is closed. JVM is reporting the same error to the driver.

Can you trying increasing the overhead memory?