I found a solution and I think it's the only possible one: If you access the file from Google Picker, then you can also download it with v3/files/download using the accessToken used for the picker. I think that Google under the cover validates downloading that precise file that you selected with the picker.

But if you would like to download any file, that you don't access with google picker, then you need drive.readonly restricted scope