i believe as per LSTM guidelines are concerned, it works on tanh activation function by default... this is to account the usage of GPU as a default computational hardware with CPU. if you change that activation, GPU will not be used. Switch back to tanh, and check again.