The audience claim is part of the payload of a JWT. Meaning it is to be set in IdentityServer4, not in your API or webAPP.
An access token must have an audience ('aud' claim). This claim tells the recipient of the token who the token is intended for.
Who the token is intended for is something that is configured on the authorization-server (in this case IdentityServer4). This is the place to configure a permission matrix. For example client a may access resource x, y, and z, in contrast to client b which may only access resource y.
The API that recieves a token must verify the audience claim, by doing so the "permissions" (being the audience and scope settings configured on each client) can be enforced.
TLDR: Your configuration in your ASP.NET Component and your SPA are correct, however the configuration on the IdentityServer is not.
Sources: