Figured out the problem. We had no control over creating the function and we found out that they had set it to .NET8 isolated worker, instead of .NET6 In-process.
Had to rewrite the code to .NET8 isolated, then it were able to run successfully.