This was a nasty one took me hours to finally figure out, in this documentation Fetch OAuth token you will notice the curl has last line data-urlencode, the line above this doesn't have new line backslash

So when i was running this curl the token was getting generated but it was missing the authorization_details

For fixing this you have to make sure that your fetch token curl is complete and looks like this

      export CLIENT_ID=<client-id>
      export CLIENT_SECRET=<client-secret>
      export ENDPOINT_ID=<endpoint-id>
      export ACTION=<action>

      curl --request POST \
      --url <token-endpoint-URL> \
      --user "$CLIENT_ID:$CLIENT_SECRET" \
      --data 'grant_type=client_credentials&scope=all-apis' \
      --data-urlencode 'authorization_details=[{"type":"workspace_permission","object_type":"serving-endpoints","object_path":"'"/serving-endpoints/$ENDPOINT_ID"'","actions": ["'"$ACTION"'"]}]'

Now this is complete and should get you the response