Agree with the comment from @Peter regarding the return value checks.

Another potentially useful debug point here would be to convert your poll() example to use the polling itself as a simple delay by changing the final parameter to 1 (meaning a 1 millisecond timeout):

else {
        pollfd pfd{};
        pfd.fd = fd;
        pfd.events = POLLIN;

        int ret = poll(&pfd, 1, 1); // final parameter is 1ms timeout

Then emit explicit debug for the values of ret and pfd.revents every iteration before you run the subsequent error checks on them.

Side note: from personal experience of similar gotchas in the past, try using static_cast<unsigned>(byte) when you log the received bytes, this can sometimes reveal interesting behaviours.