The error is due to pyspark because it is not able to deal with column names with special characters. So either replace the column names from the array or use col(f"{column_name}").

https://www.mungingdata.com/pyspark/avoid-dots-periods-column-names/