Same issue here. It looks like EF scaffolding will not use your user secrets from the CLI, unless you set environment variable "ASPNETCORE_ENVIRONMENT" to "Development".
That's what worked for me. Found in this Answer to a related question.