I use this logic in the Databricks LTS version 15.4. There I saw which is the version of hadoop: https://docs.databricks.com/aws/en/release-notes/runtime/15.4lts org.apache.hadoop --> hadoop-client-runtime --> 3.3.6
Do you expect that it should work with it?
Best regards Ruben