I have not found the actual problem. But while doing some testing I switched over to an old version of the linux kernel from TI (linux_ti_stating_6.1) and solved the issue with the UART ttyS1. I have kept the defconfig, devicetree, and everything else exactly the same, I just added the older kernel and specified I want to use it in yocto.

Why the newer upstream kernel or newer TI kernel is causing a problem with only ttyS1 and not ttyS4 is still something that is causing me confusion. I will start adding some newer kernels until I can see the issue return.