This was a databricks infrastructure bug that has now been resolved. It did not require any code or configuration change.