Apparently the issue was what I suspected in my edit. The nextjs config I had was for turbo which only works in development mode, hence the worked in dev server but not during the build. Adding a very similar config for webpack for the build solved the issue.