Don't tweak offsets by hand, get the center coordinate(s) using patch.get_center() and align the text around it by giving ax.annotate the parameter va="center" for vertical or ha="center" for horizontal alignment, respectively.

https://matplotlib.org/stable/api/_as_gen/matplotlib.axes.Axes.annotate.html