After some research today, the issue was an existing installation of Git-Bash on the remote windows machine conflicting with the new installation of cygwin. My solution was to install rsync directly into Git-Bash environment and forgo cygwin altogether. Basically following the instructions found here: https://stackoverflow.com/a/76843477/4556955

My final rsync command wound up having this format:

rsync -r -e "ssh -i my_edcsa.pem -p 55297 -o StrictHostKeyChecking=no" publish/wwwroot my-username@${{ secrets.SERVER_HOST_STAGE }}:/c/project/destination