I have figured it out. networkService.GetDetailsById(networkId)
filtered the networkUser, but did was implemented like this: network.NetworkUsers = network.NetworkUsers.Where(x => x.UserProxyId == currentUser.Id).ToList();
which overwrites the list and EF Core thinks i want to delete the rest.
Oops.