I was able to fix it by specifying the PKG_CONFIG_PATH
PKG_CONFIG_PATH
So adding this line ended up resolving it
ENV PKG_CONFIG_PATH=/usr/lib/x86_64-linux-gnu/pkgconfig