I believe the relevant settings are jobmanager.memory.jvm-metaspace.size and taskmanager.memory.jvm-metaspace.size.
How these values are used in described in the docs on the JVM Parameters.