This seems to be an problem comes woth JDK 11 & G1GC. The memory is not released back to the OS even after completing load goes down. I have experienced the exact same symptomps you mentioned while doing load tests.

Nothing wrong with Garbage collector. I checked it using jcmd and jstat commands.

This article also mentions the same issue.
https://thomas.preissler.me/blog/2021/05/02/release-memory-back-to-the-os-with-java-11

As a permanent solution, upgrading the JDK to newer version or different garbage collector can be considered.

In addition to that I was able to obtain lower memory consumption by tuning Xms JVM parameter. However this does not address the original issue but sets the memory utilization in a lower level.