Turns out I needed to add a DNS resolver to AWS DNS under the server block:
server { resolver 169.254.169.253 valid=10s; }
After that, the issue never happened again.