I am in the same situation. I need to limit the consumption speed of my Kafka consumer as the external financial API I am calling from the consumer has a rate limit setting. Plus the rate limit in my case can be different per data type.

Based on this thread, I am thinking to implement the following pattern:

1. set max.poll.records to one (1). I think I don't need to tune other Kafka parameter.

2. Create a counter in a distributed cache (e g. Hazelcast) where I save the timestamp of the 1st message that I have been received and the counter of the received msg from that time. I need to save it in a distributed cache as I use microservice architecture and I can have multiple Kafka consumer groups attached to the same topic.

3. Let's say external API can handle 3 request per a second.

4. Kafka consumers consume quickly that 3 messages and then the counter state in hazelcast shows 3.

5. After receiving the 3rd message, the Kafka consumer pauses itself before starts processing the received message. At the same time with the pause I start a sprint timer with waiting for 1 second as this is the rate limit definition. Timer schedule can depend on the data type and can be read runtime from the spring property file

6. Then when the timer fires after 1 second, it resumes the Kafka consumer.

I think this process can work. Concurrency of Kafka consumers+ different rate limit per data type can complicate the data that I need to keep in the distributed cache, but not super hard to manage it properly.

I think this way I can limit the speed of the calls of the external API. Plus if the rate limit is 3 request per one second then the 1st three data will be served quickly and then consumer(s) will wait 1 second, then continue to listen for the next data from Kafka.

I am not implemented this yet, but I will to do it soon.

I think it can work. Any thoughts are appreciated.