Please find few of the Tuning we have done to reduce the Latency in GET/put. We dont say we have made it to work like < 1 ms,but brought atleast to 50 ms to 100ms. I have mentioned thought might be useful to some one who are/were facing similar issues

Key Points

• Apache Ignite - From Embedded Server - We moved to External Ignite as Thick Client

• Choice of Thick Client vs Thin Client depends on your use case
  https://www.gridgain.com/docs/latest/getting-started/concepts

• Disable <includeEventTypes> if you are using in Configuration .xml which causes lot of communication between server nodes, Instead start using Continuous QUeries for capturing Events - This was suggested by Ignite Gridgain Experts too

• Ensure you have done Proper JVM Tuning - Sizing

• Define CustomData Region in addition toDefaultDataRegion for all in-memory data Caches
  In Memory - Means you store data in cache ,which doesnt have any Database Store attached to it.

• Instead of Put, you can Try Putall, if your application usecase supports that, it improves a lot

• Ensure you have Datasource backed with Hikari Connection Pool in Ignite configuration , for all Write Behind Write Through Caches

• Backup=1 is more than enough for 3 server node cluster

  <beans xmlns="http://www.springframework.org/schema/beans"
    xmlns:util="http://www.springframework.org/schema/util"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://www.springframework.org/schema/beans
  http://www.springframework.org/schema/beans/spring-beans.xsd  
  http://www.springframework.org/schema/util 
  http://www.springframework.org/schema/util/spring-util.xsd">                       <bean id="dataSource" class="com.zaxxer.hikari.HikariDataSource">
        <property name="driverClassName" value="oracle.jdbc.OracleDriver" />
        <property name="jdbcUrl"
            value="jdbc:oracle:thin:@**********" />
        <property name="username" value="<<YOur Schema User>" />
        <property name="password" value="**********" />
        <property name="poolName" value="dataSource"/>
        <property name="maxLifetime" value="180000"/>
        <property name="keepaliveTime" value="120000"/>
        <property name="connectionInitSql" value="SELECT 1 from dual"/>
                <property name="maximumPoolSize" value="20" />
        <property name="minimumIdle" value="10" />
        <property name="idleTimeout" value="20000" />
        <property name="connectionTimeout" value="30000" />
  
    </bean>
  <bean
                    class="org.apache.ignite.configuration.CacheConfiguration">
    <property name="name" value="YOURCACHENAME" />
    <property name="cacheMode" value="REPLICATED" />
    <property name="atomicityMode" value="ATOMIC" />
    <property name="backups" value="2" />
    <property name="cacheStoreFactory">
    <bean class="org.apache.ignite.cache.store.jdbc.CacheJdbcPojoStoreFactory">
  <property name="dataSourceBean" value="dataSource" />  <!-- Mention Datasource Bean-->