This may not be relevant to big query though I assume a similar workflow is possible you may want to look into. In Snowflake we use the GCS storage integrations which allow you to basically query csvs inside of the gcs buckets. We were able to then setup a script that would copy the data from the file into a temp table, then run a similar merge script like you have here to quickly merge (upsert) data into our true sources then drop temp table. Works like a charm and minimizes compute time.

I'd recommend trying something like that to see if it works for your purposes.

use CSV helper to take your list and turn it into a csv memory stream:

https://joshclose.github.io/CsvHelper/examples/writing/write-class-objects/

upload memory stream to gcs bucket

https://cloud.google.com/storage/docs/uploading-objects-from-memory

move data from gcs to bigquery into temp table

https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-csv#c

now run your merge script.

cleanup any files/temp tables