The following worked for me, Apache Spark 3.5.5 is compatible with Python 3.8 to 3.11 (Python 3.11.9 is recommended). Also make sure to add PYSPARK_PYTHON=python as an env variable.