I managed to remove this error by cropping the image and giving each one a .box file and a .gt.txt file. For the x values I could just use one image

My original Image:

My cropped image for x values:

For the y values, if my cropped image contained anything of the graph my training would fail. So I just cropped the value. Here is an example:

I still haven't figured out how to train tesseract without modifying my image.