To answer my own question: I got the bright idea of using SMS. Google Assistant can send them. I have an Arduino with a number that can receive them, and if they are from my number they can get passed through using a POST request. Not the most secure solution in the world, but good enough for my personal needs.