TL;DR: It seems like a recent bug in AWS Cognito new release. To workaround this, switch to the old hosted UI in Cognito.
I will answer myselft if someone encounters this same issue. It looks like a bug in Cognito new hosted UI, as they have release a new version in 22 Nov. 2024 with a new hosted UI.
If you are using the new Cognito UI with ALB authentication, it will return 401 page when a user is signing up for the first time. This only happens on sign up, then if you log in again in a new tab it will work well. If you switch to the old hosted UI in Cognito, the issue is gone.
I hope they fix it soon as this was released a couple of weeks ago.