To Solve This you need to rebuild the setup.py specifying the --rpath flag (-R)
Like this :
python3 setup.py build_ext -i -R <libdirectory> -f
the -f force the build