In spark 3.5, they introduced two equality test functions for PySpark DataFrames: assertDataFrameEqual and assertSchemaEqual. An overview can be found at this link https://www.databricks.com/blog/simplify-pyspark-testing-dataframe-equality-functions