For installs done in conda environment,

pip uninstall package_name

then delete the .egg-link file in
path_to/miniconda(anaconda)/envs/<env_name>/Lib/site-packages/<package_name>.egg-link