Found the issue !
My Postman HTTP version in the settings was set to Auto, which I guess uses HTTP 2.
When I hard select HTTP 2 it indeed comes back without the port, but when I put it on HTTP 1.x the port is included... Go figure...