Try implementing Dropout and Regularization while defining the model, your model might be concentrating more on non-zero pixels.