In my case I had to add listen [::]:80 for ipv6 as well. After that - it works.

server {
    listen 80;
    listen [::]:80;
    server_name my-site.com;

    root /var/www/html;
    index index.html;
    .....
}