I've run into this and found the problem to be a ton of connections from random servers trying to login. What helped was using a different port like 5960, anything that isn't the default port 5900. It seems people are out there hammering that port.