Setting spark.driver.extrajavaoptions and spark.driver.extrajavaoptions to -Dlog4j2.configurationFile=<path to my log4j2.properties file> -D-Dlog4j2.debug worked for me. The steps on the page https://logging.apache.org/log4j/2.x/faq.html#troubleshooting are quite helpful.