Should I sync /public and /.next/static with an S3 bucket in the github workflow?

Yeah this would be a sensible deployment option, offloading the serving of static assets to S3 / CloudFront. Another option would be to put CloudFront in front of your Next app and let CloudFront cache based on cache control headers. Less optimal in terms of reducing requests to Fargate, but a slightly easier deployment.

If so, should I build the nextjs app once in the github workflow and then deploy different parts of the same build to both fargate and S3?

Yeah ideally build the app just once. You could do this in a separate step, then copy the relevant artifacts into your image, rather than building the NextJS app in the docker build. Then copy the static bits up to S3.

Are there any best practices or github actions that already do this out of the box?

Don't know about github actions, but you could take a look at how OpenNext does it (in terms of splitting the app and deploying). I imaging there will be some complexity around keeping the 2 parts in sync during rolling updates, roll backs, deleting old files etc. I've done something similar and my approach was to deploy everything to a subdirectory in S3 then switch the CloudFront OriginPath once everything is deployed.