For future readers, Postgres has a built in solution, REPLACE().

Here's it added to my CONCAT() line to accomplish the desired result:

REPLACE(CONCAT({concat_columns}), '"', '')