You use libx264rgb, however mileage may vary depending on how you plan to play it (for instance if I watch it in mpv it looks correct, but in chromium the colors looked very much wrong). libaom-av1, libvpx-vp9 and libx265 also support this via the gbrp pixel format.