In my case, I was able to get rid of the error by upgrading my spark version from 3.3.1 to 3.4.0
3.3.1
3.4.0