I think you should add always to ensure the header is sent even on error responses.

add_header Content-Security-Policy "upgrade-insecure-requests" always;