I have waited for periods longer that 5mins for scale down which never tirggered, I then deleted the hpa object, scaled replicas to 4 and recreated the hpa. After few minutes it scaled up to 8, and after another 5 minutes it scaled down to 7. So now it is working as expected, I have no clue why it was "stuck".