Reports

Speaking about your question:
"expect libraries to be loaded before applying the configuration?" - it is correct, but for libraries related to cluster, not the user application libraries.
That means, you need to look into initialization script for cluster. (https://docs.databricks.com/aws/en/init-scripts/)
You should place the load of your desired extensions in the script, for instance like here (init.sh):


DEFAULT_BASE_PATH=""
BASE_PATH=$1

DB_HOME=${BASE_PATH}/databricks
SPARK_HOME=${BASE_PATH}/databricks/spark
SPARK_CONF_DIR=${BASE_PATH}/databricks/spark/conf
SPARK_JARS=${BASE_PATH}/mnt/driver-daemon/jars

setUpBasePath() {
if [[ DEBUG_MODE -ne 0 ]]; then
    logInfo "Init script is going to be run in local debug ..."
    logDebug "Check if BASE_PATH is provided for debug mode."
    if [[ -z ${BASE_PATH} ]];then
        logDebug "BASE_PATH is unset for debug mode. Please provide it."
        exit 1
    else
        logInfo "Arg BASE_PATH is provided: $BASE_PATH"
    fi
else
    logInfo "Init script is going to be run ..."
    BASE_PATH=$DEFAULT_BASE_PATH
fi
}

setUpBasePath


# Init databricks utils
curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
STAGE_DIR=$(mktemp -d)
#databricks workspace export-dir /Shared/Init/15.4_LTS $STAGE_DIR --overwrite
${HOME}/bin/databricks workspace export-dir /Shared/Init/15.4_LTS ${STAGE_DIR} --overwrite --debug
ls -R ${STAGE_DIR}

logInfo "Copying listener jars..."
cp -f "${STAGE_DIR}/libs/spark-monitoring_15.4.0.jar" ${SPARK_JARS} || { echo "Error copying file"; exit 1;}
curl https://repo1.maven.org/maven2/org/apache/logging/log4j/log4j-layout-template-json/2.22.1/log4j-layout-template-json-2.22.1.jar > ${SPARK_JARS}/log4j-layout-template-json-2.22.1.jar || { echo "Error fetching file"; exit 1;}

So, as you can see above, I am using Databricks CLI to place the necessary jars (consider them as reference, you need to manage jars and their path by your own) into Databrciks cluster driver folder during startup.

After initscript execution, configuration that you have in cluster config should work.
Also, I guess you could do it manually by putting extension jars in DBFS folders that are service folders for cluster, for that purpose check the global variables in the provided code snippet.

79519394