Ok something weird happened. I'm pretty sure it got stuck in a weird loop.

I removed the Cloudflare DNS record and removed the custom domain from GitHub.

Added it again on Cloudflare and then GitHub pages and it started working fine