For anyone who is still getting the error after granting access. I tried to delete the key vault secret reference from my app service's environment variable, save and re-add it back, and it works now