So the implementation idea is:
Valkey stream–based queue with a threshold for batch processing.
The producer adds tasks to the stream (using XADD
).
The consumer checks the stream length (using XLEN
) and reads tasks from the stream via a consumer group (using XREADGROUP
).
Once the threshold is reached, tasks are processed in a single batch and acknowledged (using XACK
) to remove them from the stream.
For the fun, created both for python and node the examples of usage, using valkey-glide client, and created a repo: https://github.com/avifenesh/glide-consumer-supplier-queue