Thank you @AmyDev for the link to the gitlab issue.
Turns out that for some reason it doesn't work with project tokens but I had success now using a personal access token (User Settings/Access tokens).
Still don't know why it didn't work with the project token and also not with my real user. Maybe 2FA was the issue with the later one?