If you're building using the cloud build default worker pool, you need to add access to the cloud build iam.
gcloud projects add-iam-policy-binding PROJECT_ID \
--member="serviceAccount:[email protected]" \
--role="roles/cloudbuild.serviceAgent"