I am interested how this answer has worked out over time since google follows robots.txt options loosely. It seems like this continues to be an issue for RTD users. Any updates? From a cursory google of pyngrok, it seems like it works well.