Problems like this often come from version incompatibilities between Spark, Python, Java, or Hadoop. In my case, the following combination works without issues: Python 3.11, Java 17, Hadoop 3.3.6, and PySpark 3.5.1.