I had the same issue. Noticed in AWS Console > Hosting > Rewrites and redirects, by default there was a redirect for <*> to /index.html, with type 404.
I simply changed the type to 200 and this fixed the issue.