I managed to get it working by excluding `Microsoft.Bcl.Memory` from the project:
<ItemGroup>
<PackageReference Include="System.Linq.AsyncEnumerable" Version="10.0.0" />
<PackageReference Include="Microsoft.Bcl.Memory" Version="10.0.0">
<ExcludeAssets>all</ExcludeAssets>
</PackageReference>
</ItemGroup>
Don't know why I need to do this though.