The solution was to set the whole chartData another time after setting the new values for the annotation. I don't know why that is working and why @naren muralis example doesn't but it works now for me.