After a day of fiddling what fixed it for me was:

ports:
      - "5434:5432" # So 5434 for local host, but still 5432 in the container

and then access by:

psql -h localhost -p 5434 -U root -d users