If you are writing a library, a class not being referenced within your project does not imply it won't be used by your users.

Write unit tests and set up a test coverage report. Then you will have a better idea whether those uncovered lines of codes are actually "unused" in your situation.