Try setting this in your gpg-agent.conf.
gpg-agent.conf
default-cache-ttl 2592000 max-cache-ttl 2592000 allow-loopback-pinentry