After hours of struggling, I found out the solution by accident. All I had to do, was to add this configuration:
grpc: shutdown-grace: 30