I found the issue.

I was running the server from VSCode. Running in a mac terminal fixed it.

I think it had to do with VSCode settings blocking network connections.

When I restart my mac, it restarted vscode and restarting vscode works the first time. I still don't know why.

Quick fix: don't use VSCode to run the server. It has it's own network settings