I found the issue.

I misinterpreted the docs and assumed that if you changed the ports in de docker-compose, you changed the ports on the container. This turns out to not be the case. So while my host forwarded its 8085 port to RabbitMQ's 5672, the container itself still had 5672 as its port.

Changing the port mapping to 5672:5672 fixed the issue.