Maybe something like this?

var users = Users.ToAsyncEnumerable()
    .WhereAwait(async x => await IsGrantedAsync(x.Id, permissionName));

This uses the System.Linq.Async package as you cannot provide an asynchronous function to the Where method.