I have found out that if I removed the signing type from the authentication sub key (which was enabled for authentication and signing) it starts to work... I have no idea why git/gpg choses to use the sub key instead of the master key and why using the sub key doesn't work.