So I did mention "But I want to have it work without this port mapping.". What I meant I don't want to use this mapping on the RabbitMQ container, since then it bypasses the Traefik.

So dumb me forgot to add port mapping to the Traefik docker-compose file. Added the port mapping 5672:5762 to the docker-compose.yml of Traefik and I can connect over telnet to the server.