Can you try to force the permission by overriding get_permissions in your viewset (perhaps also you can specify AllowAny for particular actions)?
get_permissions
AllowAny
def get_permissions(self): return [AllowAny()]