I was able to solve a similar issue by deleting:

~/Library/Caches/org.swift.swiftpm/repositories/<folder for package>

and

~/Library/org.swift.swiftpm/security/fingerprings/<json file for package>

And then re-adding the package to amy project.