For anyone still looking for a solution, the fix I found was to assign the shortcut to a button on my Razer mouse with the Razer Synapse app. In Synapse, click on the button you want to change, select "Launch" and select the "Website" option. Paste the filepath from your Windows shortcut into the field (e.g. "%SystemRoot%\system32\WindowsPowerShell\v1.0\powershell.exe -ExecutionPolicy Bypass -File "C:\Users\kekus\Documents\scripts\audio_switcher.ps1"). Save.
For whatever reason, the startup time is reduced to a few milliseconds.