If anyone runs into the same problem:

The only solution I found was by switching to Azure Flexible Consumption plan to allow for vnet integration and then using a vnet / service endpoint to let the Azure Function access KeyVault secrets.