I found a solution based on this post:
64 bit function returns 32 bit pointer
turns out I needed to define the function which calls the mock().actualCall in my header file of the c function