As @gaston (I don't know how to @mention people here) pointed out and as I half suspected there were versioning issues as with all the other instances we see of this issues on SO. what I did to resolve it was
So now the project will compile with all the latest available for Spark 3.4
However that brings me to my next blocker..... (barf)
2024-10-30 17:14:14,980 ERROR ApplicationMaster [Driver]: User class threw exception: java.lang.NoClassDefFoundError: scala/collection/ArrayOps$
I'll be investigating this now, and perhaps another SO post..