What is your deployment environment? Are you using dedicated search nodes? Or the coupled architecture? And could it be related to this issue where readPreference=secondaryPreferred appears to affect pagination?
readPreference=secondaryPreferred