Given that one wants to use the Spark configuration to do this check - and use secrets kept there one can use the following code :

val fs = FileSystem.get(new URI(<url to blob storage>), spark.sparkContext.hadoopConfiguration)
val path = new Path(url + s"/$directory")
fs.exists(path)