This turned out to be a duplicate of "AWS CloudFront access to S3 bucket is denied". The "Origin Access Identity" "Signing behavior" must be "Always sign requests". I had several origin access identities and I somehow picked the one with no signing. You can view and edit your Origin Access Identities in the cloudfront console at the top level under Security > Origin access.