What is your podman machines memory set to? (Assuming you are using podman with podman-compose).

Check with

podman machine list

If the memory is set to 4GB then I believe that is the limit a single container can have regardless of configuration.

You can try recreating the machine with a higher memory allocation. Setting to more than 4GB should allow you to exceed the 4GB limit you are observing on your containers, up until the limit set by the machine.

podman machine init --memory=8192