I think problem is in SRARK configuration. Add, pls, PYSPARK_PYTHON environment variable in your ~/.bashrc. In my case, it looks like : export PYSPARK_PYTHON =/home/comrade/environments/spark/bin/python3, where PYSPARK_PYTHON is path to my python executable in "spark" environment.1

Hope, it helps)