After searching for a long time, we found that Wix 4 changed the way component guids are generated, so they are not stable across versions. This leads to the above problem. Wix 6 has a fix for this, making it use the previous way of generating the guid:

-bcgg command-line switch

Wix bug ticket: https://github.com/wixtoolset/issues/issues/8663