you can use these 3 endpoints in conjunction

    url = f"{DATABRICKS_INSTANCE}/api/2.0/jobs/runs/list"

    url = f"{DATABRICKS_INSTANCE}/api/2.2/jobs/runs/get"

    url = f"{DATABRICKS_INSTANCE}/api/2.2/jobs/runs/get-output"

1. getting list of jobs

2. providing task run id

3. processing task run id and error message

i am filtering based of result_state != success as well

were you able to get a better solution?