In order to make spark read the log4j.config file follow below steps:

1. Add log4j.config file in root directory in your local
2. Go to Spark directory: search for conf folder and add the following to spark-defaults.conf.template: spark.driver.extraJavaOptions -Dlog4j.configuration=file:log4j.properties -Dspark.yarn.app.container.log.dir=app-logs -Dlogfile.name=hello-spark

in the above example I app-logs refer to the directory where my application logs will be collected and hello-spark is the name of the log file. change these according to your use case. It works!!!