I solved this problem for my employer who is heavy into protobuf and wanted to use it for file formats.

https://medium.com/@unclepaul84/efficiently-reading-and-writing-very-large-protobuf-files-local-disk-and-s3-approaches-a289c8855606