So me not being able to SSH to my own server using the public IP address was indeed a NAT loopback/hairpin issue. All the online resources to test SSH servers would show that my server is accessible, and I managed to complete the challenge by changing the default shell of OpenSSH to Bash.

"New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Program Files\Git\bin\bash.exe"