I did follow ssh-copy-id and all however it was still expecting password so added below entry to defaults section of ansible.cfg that helped to avoid mentioning --ask-pass and typing password.

ask_pass = no