I had to give the sender permission at the Event Grid Domain level to resolve this issue. I was myself expecting the granular permissions to be at domain topic level as per the docs. However this does not seem to be the case. With the top level permissions in place i was expecting it to fail when i removed the topic level permissions but it didn't.

Apologies this is only answer, I know of at this time.