If you can revisit the documentation for the api that you are using to download a file, you will see that it requires either of the three Authorization Scopes but reading further at this documentation we can see that the scope that is meant for downloading is https://www.googleapis.com/auth/drive.readonly which is restricted, and such scopes requires authorization for security reasons
TL;DR
It is not possible to bypass authorization of restricted scopes when downloading files, in Security perspective, you would not want anyone to just download your file in Google Drive.
One approach you can possibly take is to file a Feature Request though again, this is a security risk and most likely will not get that much attention but it is worth trying.
References: