This is caused by a case-sensitivity issue, foo and Foo.
This can be resolved by adding
.config("spark.sql.caseSensitive", "true")

It will treat foo and Foo as different columns.