It seems display/show command by default add limit when query is executed to database.

Try using df.collect()

#Convert List[pyspark.sql.types.Row] to Spark data frame

df=spark.createDataFrame(df.collect(),schema=df.schema)

display(df)