What I was missing was to set ZBX_SERVER_HOST to the name of the service defined in the docker-compose file.

Maybe I am missing a point but I believe ZBX_SERVER_HOST should be present in the docker-compose example because the default localhost that ZBX_SERVER_HOST goes to will result in a correct connection in most containerized configuration of Zabbix web server + Zabbix server.

In my case the relevant part of the docker compose is:

services:
  zabbix-server:
  ...
  zabbix-web:
    environment:
      ...
      - ZBX_SERVER_HOST=zabbix-server