In spark 3.5, they introduced two equality test functions for PySpark DataFrames: assertDataFrameEqual
and assertSchemaEqual
. An overview can be found at this link https://www.databricks.com/blog/simplify-pyspark-testing-dataframe-equality-functions