I tried everything. Nothing actually worked reliablly. I was using an AdafruitDisplay M4 Matrix on the other end of the C# script. I found that if I use serial.read, I see this issue. Changing the method to serial.readline() solved my issue related to semaphore time out errors.