Got the solution from someone! I had to add a runtimeconfig.template.json to the C++/CLI project with this property set:

{
  "configProperties": {
    "System.Runtime.InteropServices.CppCLI.LoadComponentInIsolatedContext": true
  }
}