When using Service Accounts for authentication then the sharing settings for the service account within Google Calendar must be set to at least "Make changes to events" so "See all event details" is not enough to get colorId.