79823966

Date: 2025-11-19 00:15:55
Score: 0.5
Natty:
Report link

We use celery queues for long running test tasks, so we want only one task per worker. After fighting this same problem using task_acks_late but getting occasional unwanted prefetches, I did some code spelunking and additional testing using much newer versions than discussed in this thread. (Celery 4.4.7 and 5.4.0 using rabbitmq-server 3.12.11).

I find the following:

As noted in David Wolever's reply,
worker_prefetch_multiplier = 1
does not stop prefetching. It really means "prefetch 1 in addition to the already running task".

As observed using RabbitMQ monitoring (http://myhost:15672), setting
worker_prefetch_multiplier = 0
does indeed seem to stop prefetching. More long-running testing is needed to verify, but reviewing the code shows the prefetch value gets passed (more or less) directly to the RabbitMQ qos calls as prefetch_count. Setting it to zero does not appear to "allow the worker to keep consuming as many messages as it wants". Rather, it sets the qos prefetch_count to zero.

The documentation (https://docs.celeryq.dev/en/stable/userguide/configuration.html) is incorrect about both of these setting values, at least with RabbitMQ.

Reasons:
  • Long answer (-1):
  • No code block (0.5):
  • Low reputation (1):
Posted by: J. Reuter