I'm not sure this will fix things but have you just tried to send the GroupID by itself?
mailMessage.SetAsm(unsubscribeGrpId)
Also, I've noticed if you don't have a valid group ID (which you find in the SendGrid Web Dashboard -> Suppressions -> Unsubscribe Groups), it will not give you an error but the email will not be sent.