For bit-identical reproducible builds, I found via https://wiki.debian.org/ReproducibleBuilds/OggSerialNumbers that you have to call oggenc -s $number $input to override the random serial number in the bitstream.