I managed to get it working without using workerSrc. I only needed the following import:

import 'pdfjs-dist/build/pdf.worker.mjs'

This is also mentioned in a comment on the react-pdf github issue