Setting spark.driver.extrajavaoptions
and spark.driver.extrajavaoptions
to -Dlog4j2.configurationFile=<path to my log4j2.properties file> -D-Dlog4j2.debug
worked for me. The steps on the page https://logging.apache.org/log4j/2.x/faq.html#troubleshooting are quite helpful.