This problem is persistent even in new version upgrades of 3.5.x I just go 2 versions back and everything was ok.spark-core_2.13 and 3.5.0 is good for me