The issue with this was that it was a Local Service fabric application and Service Fabric realm doesnot recognize beyond the SfAppCluster level, hence was not able to pick up the Visual studio credentials

What we did was to go the service principal route and set our environment variables ( https://learn.microsoft.com/en-us/dotnet/azure/sdk/authentication/local-development-service-principal?tabs=azure-portal%2Cwindows%2Ccommand-line )

And placing the S_+NI cert on the SfAppCluster folder localy