I found the problem was with within the project code itself. (I'm not the maintainer of the code base, and it is quite large.)

One of the differences was a git commit hash that was integrated into the binary. When I was verifying that the binaries were equal, I was compiling the code on different commits. Therefore the git commit has was different.

The other difference was the timestamp. This was embedded in the binary because a third-party library was using the `__TIME__` macro.

So if others run into the same issue, looking into similar things might be the solution :)