It is a compatibility issue between Avro and Spark. Remove from the spark.jars.packages, org.apache.spark:spark-avro_2.12:3.5.3 since in the Spark latest vesrion already included. Secondly use latest Hadoop-Azure version "org.apache.hadoop:hadoop-azure:3.3.4". Refresh your spark session, it will resolve the issue.