I've never encountered such scenario but try using sshpass -p <password> ssh <username>@<ip-address> at the beginning and see if it helps.