I was finally able to understand where the difference is coming from. I was using GPU for Tensorflow/Keras so the computations are indeed different from Numpy, which runs on CPU.

Using this to have Tensorflow/Keras running on CPU got me the same result as in Numpy:

import os
os.environ['CUDA_VISIBLE_DEVICES'] = '-1'