I found a solution, there two back pressure threshold you can set for a flowfile queue. The object threshold: # of flowfiles size threshold.

In my use case, it is the object threshold got hit almost all the time. I set this threshold to 0 which will be unlimited. Now the back pressure will only be triggered if size limit is reached. Size limit is hard to reach (there's still a risk though) in my flows.