The question is about version 3, but now that the modern docker compose replaces version 3 I will provide an answer for that.

The new compose spec (without any version) has a simple mem_limit option:

services:
  image: example
  mem_limit: 1G
  cpu_count: 1

See: https://github.com/compose-spec/compose-spec/blob/main/spec.md