Just don't set your GPU's memory growth. If you have any kind of code like as follows, don't run it, it should work fine:

#Preventing Out of Memory error for the GPU
gpus = tf.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
    tf.config.experimental.set_memory_growth(gpu, True)