Thanks @mklement0 for the great links. That was a set of good reads to refamiliarize myself with.

It turned out in my case that I needed to surround the password with " characters.

I failed to mention that I am calling az postgres flexible-server execute in a PS1 script that is itself called from another PS1 script, all running in pwsh.

Ultimately what worked was, basically:

# $rawPW is the raw password from a KV Secret or elsewhere
$p = """" + $rawPW + """"
...
az postgres flexible-server execute -n the-instance-name -u theName -p $p -d theDatabase -q "select * from MyTable"