Recently published a blog trying to explain the above https://medium.com/@ankush13777/the-hidden-optimization-behind-sparks-mappartitions-28983541df18