Create a Snowflake TASK to execute the COPY INTO command and execute that task from your process. This runs the COPY INTO commands serially as needed, eliminating the concurrency issue. Per the EXECUTE TASK documentation:
- Snowflake ensures only one instance of a task with a schedule is executed at a time.
- If the root task is currently running (i.e. in an EXECUTING state in the TASK_HISTORY output), the EXECUTE TASK command schedules another run of the task to start immediately after the current run is completed.
- If the root task is currently scheduled (i.e. in a SCHEDULED state in the TASK_HISTORY output), the scheduled run is replaced with the requested run as usual, with the current timestamp as the scheduled time. However, if the scheduled time has passed (but the task has not yet transitioned to an EXECUTING state), then the scheduled run occurs as usual. That is, the scheduled run is not replaced with the requested run.