I wanted the images to load up in both flask app in production and in react app (using npm start) for development. At the same time, I wanted unknown pages to be handled by react.
The most convenient solution I found is to put all the images under public/static/images instead of public/images and to change the src paths accordingly.
Then I can run yarn build and get the result build folder to the flask app folder (backend). That worked perfectly.