It turns out the original variable assignment was the problem. I should have used single quotes instead of double quotes. This code:

$Passwd = 'StringWith`Character'      # <-- entered by user
$new_pw = $Passwd.replace('`','``')
write-output "Passwd = $Passwd"
write-output "New_pw = $new_pw"

Produces this output:

Passwd = StringWith`Character
New_pw = StringWith``Character