You should change the code

from

    .config("spark.hadoop.fs.defaultFS", "hdfs://localhost:8020") \

to

    .config("spark.hadoop.fs.defaultFS", "hdfs://hdfs-namenode:8020") \

Because from your hadoop hdfs-datanode can't access hdfs://localhost:8020, it should be hdfs://hdfs-namenode:8020.