The gems were installed to the default GEM_HOME directory. This is likely /Library/Ruby/Gems/ on a mac. You can go in there and manually rm -rf the stuff in there without worrying.