The code is absolutely fine. I suspect the issue is due to incompatibility between the Spark version you are using and the Python version you have. Please downgrade to PYTHON3.12 and see if that helps.