For my concrete problem, I could solve the issue by "just switching" to esbuild:

-      "executor": "@angular-devkit/build-angular:browser",
+      "executor": "@nx/angular:browser-esbuild",

While I don't understand the underlying issue completely, I am fine with "the fix".