After debugging, I found how I constructed the $endpoint with variables is troubling. Not the permission issues.

Below is the correct one without single quotes:

$endpoint = "https://us-central1-aiplatform.googleapis.com/v1/projects/$project/locations/$location/publishers/$publisher/models/$model:predict";