I've been suffering from this problem for a while now too.

When configuring Spark settings such as 'spark.scheduler.allocation.file' in Databricks Runtime 16.4, it is important to specify the correct file path and ensure proper access permissions. The error message 'java.lang.RuntimeException: Failed to get WSFS info. Missing token.' indicates that the system is unable to authenticate or access the specified file. In Databricks, files stored in the workspace are typically accessed via DBFS (Databricks File System) rather than direct file paths. You should upload your 'fairscheduler.xml' file to DBFS, for example using the Databricks CLI or UI, and then reference it with a DBFS URI, such as 'dbfs:/path/to/fairscheduler.xml'. Additionally, ensure that your cluster has appropriate permissions and that any required tokens or authentication methods are correctly configured. For example, you might set the configuration as follows: config("spark.scheduler.allocation.file", "dbfs:/init/fairscheduler.xml") This approach ensures that Spark can locate and access the configuration file properly within the Databricks environment.