you have set `ssh_agent_auth` to true, have you started ssh agent in the machine where you are running your packer build.