I ended up going with this:

increase((count by (repo,workflow) (github_workflow_run_status{repo='xxxx/yyy', workflow='zzz'}) - sum by (repo,workflow) (github_workflow_run_status{repo='xxx/yyy', workflow='zzz'}))[2h:5m])

This gives me the number of failed jobs in the last two hours to work with for my alert.