It seems that your model is predicting all the numbers as one number, so I would recommend checking your loss function. I suspect you might be handling the true labels incorrectly.