pandas data frames use eager executon model by design

https://pandas.pydata.org/pandas-docs/version/0.18.1/release.html#id96

Eager evaluation of groups when calling groupby functions, so if there is an exception with the grouping function it will raised immediately versus sometime later on when the groups are needed

The alternative is pandas on Spark - https://spark.apache.org/pandas-on-spark/

pandas uses eager evaluation. It loads all the data into memory and executes operations immediately when they are invoked. pandas does not apply query optimization and all the data must be loaded into memory before the query is executed.

It is possible to convert between the two - to_spark/to_pandas.

Similarly it is possible to convert between pandas and traditional Spark data frames - createDataFrame/toPandas.