The answer (a very comprehensive one) is here: https://stackoverflow.com/a/64771432/4582204 (and this question will likely be closed as a duplicate).

But I wanted to simply and quickly say that in

-replace regex, replacement

the regex is a regex, and needs the two backslashes \\ because the first has to escape the second, but the replacement is literal, no escaping needed, so \\\\ is not two escaped backslashes but literally 4 backslashes.

So the correct code is:

'foo\bar' -replace '\\', '\\'

And while at first glance this code appears to be doing nothing, it is doing something, because the first backslash is an escape and second term is not escaped.