It's likely that you've encountered a problem described here: asymmetric association updates

Make sure that you update both sides of the association when making this kind of an update.