I just realized that this is most likely because the loss is being calculated on the batch level, and I'm using a small batch. I think it makes it a rather bad question after all, but I'll keep it here in case its useful to anyone.