I would recommend using a docker environement. See this https://github.com/varunvilva-kickdrum/hadoop-hive-spark-jupyter-docker