no library will be good, so craft all encoding by hand or patch some library

Well... for this case, you need a deep knowledge of how ffmpeg works. I found a free book on Google search.

use library that allows such low level manipulation of key frames metadata

There is an example of how to parse MP4 file in Rust: dump-frames.rs