Ok, found the problem. We installed DacFX on the target machine, but it turned out to be an older version. Once we used the same version as was installed on the build agent we were able to continue with the deployment.

So if anyone here has the same issue: Install DacFX on the TARGET machine as well as on your build agent. DACPAC files need to be copied to the target machine as SQLPackage executes the deployment on the target server, not from the devops agent.