Thanks for all the useful remarks,
Seems like std::bitcast is the right way to do this. It requires C++20 but I think I'm ok with that.