In case it helps anyone, this error was solved by setting the AbstractFileSystem for gs URIs per (this)[https://github.com/GoogleCloudDataproc/hadoop-connectors/blob/master/gcs/INSTALL.md].

.config("spark.hadoop.fs.gs.impl", "com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem")

I have no idea why downloading the JAR file from https://storage.googleapis.com/hadoop-lib/gcs/gcs-connector-hadoop3-latest.jar works though