Are you using a private key for ssh? If so try to give group and others reading rights.

e.g. chmod go+r ~/.ssh/myprivatekey

Worked for me.