To get integer base values you have to reduce the y-ticks to those locations. If you now your exponent a priori, this is a one-liner. Try set_major_locator() with MultipleLocator and set it to your exponent.

import numpy as np
import matplotlib.pyplot as plt

x = np.linspace(0, 10, 100)
y = np.exp(x)

plt.plot(x, y)
plt.ylabel("y")
plt.xlabel("x")
plt.title("Exponential Function")

# Set the y-axis to use scientific notation
plt.ticklabel_format(axis='y', style='scientific', scilimits=(0,0), useMathText=True)

# Use only tick positions that are multiples of 1e4
plt.gca().yaxis.set_major_locator(plt.MultipleLocator(1e4))

plt.show()