By adding dont_filter=True in Request function, it does continue fetching without closing.

yield scrapy.Request(new_url, callback=self.parse, dont_filter=True)