Ok, fixed my problem by:
Call the api directly (so no more @googlemaps/js-api-loader).
like stated I am using the angular http client but for this request I switched to the capacitor http client (with angular: cors error response from the api endpoint)
Hope this helps:)