I was able to sort it out, my ng config was not the issue, it was the Github workflow file.

Event if a explicity specify a build command:

npm run build:qa

Apparently, internally, Azure SWA still builds the app internally using the default config (npm run build)

So I had to add this line in the deployment step:

skip_app_build: true

And also ensure that app_location points to the build output location and that ouput_location is empty.