Posting the answer as an answer because it took me a while to notice it in the OP's comment:
Simply add .AsNoTracking() before .ToListAsync()!
.AsNoTracking()
.ToListAsync()
This made my code work perfectly. Thanks!