Managed to fix this issue - it was caused due to the service connection in Azure DevOps being setup as a Federated Workspace which is the way Microsoft are trying to push you when setting up new service connections in ADO. What they don't tell you is that the OpenID token which is used by JWT only last a max of 60 minutes. So for production workloads this can be cumbersome.

The fix was to create a new service connection using a client secret, however this means secret rotation will be manual again.

If you're creating RSV via Terraform using ADO CI/CD pipelines you're best off creating your service connection from the beginning as a service principal using secret, RSV will take hours to complete the VM replication.