I also faced same issue and mapreduce.input.fileinputformat.list-status.num-threads helped.
For 50000 xml files, it was taking 13 mins but with this property set to 50, it took 15 seconds. All of this happens in the driver