I don't think this is supposed to happen. Unfortunately, the only thing that works for me is to remove the remote repo and add it again. That way, it clears the cached branches.