Instead of to_timestamp try date_format
to_timestamp
date_format
https://spark.apache.org/docs/latest/api/python/reference/pyspark.sql/api/pyspark.sql.functions.date_format.html?highlight=date_format#pyspark.sql.functions.date_format