Documentation saying "update the PR title with correct Jira ticket". Funny that when PR was merged, I have not seen PR tab any longer. But it's possible to find PR tab even for merged pull requests. So I did it, changed PR title, and it started to work. Easy fix, took long time to find though. Even if you don't see the tab, then use direct URL with PR # to get to it. And they you will be able to edit the title. For example use URL:

https://github.com/yourname/yourproject/pulls
https://github.com/yourname/yourproject/pull/3