After doing a lot of research, I realized that the issue has to do with the use of LSTM.

LSTM and RNN are critized for begin bad precisely at predicting future values of a sequence and often used for predicting intermediate values in voice recognition or sentiment analysis.

Futher research showed me that, for forecasting, it is recommended to use Seq2Seq models like an LSTM encoder-to-decoder or attention based models that don't rely on autoregression.