I am not really sure what you are trying to achieve (yes, low latency but what is the project about and why do you start with off-heap management?)

Take a look at Quarkus+Vert.x which is already super fast and saves a lot of memory - you can even step up the game by native compilation. If you still need to improve things, use specialized libraries like Eclipse Collections. It it a highly optimized Java collections library with even more types, primitive collections.

I used this combination several times to create high performant low latency applications.