I was still get prompted to enter my credentials even when connecting via SSH as @torek suggests in the accepted answer.

Then found out that when I created my SSH key I entered a password thinking I needed it for Gitlab authentication. Turns out this password was only to use the SSH key and would cause me to be asked for it on every pull, fetch, push etc.

What I did was delete the ssh key, remove it from Gitlab and create a new one following the steps in the official docs. This time, when creating the ssh key and being asked for a passphrase, I'd just press Enter to not use one.

And voilà, I stopped getting asked for a password all the time.