In my case I had a similar issue when I tried using

spark.conf.set("fs.azure.account.key.<storage account>.dfs.core.windows.net", 
               "keys")

I got the erorr SQLSTATE: 42K0I. Tried everything, the only solution that worked was I switched from serverless compute to classic compute.