You appear to be running into a typical schema evolution problem when using Protobuf and Spark. Instead of raising an error, the scalapb encoder should ideally handle missing fields gently by changing them to None. It could be worthwhile to see if the sparksql-scalapb library has any configuration changes or settings that could fix this situation. For additional information or possible solutions from the community, think about submitting a problem on their GitHub page. I wish you luck!