It's a good idea to simplify and break down the problem into smaller chunks. You can remove/comment out parts of the code and see if the problem is still there. One good candidate here is the noise you're adding - torch noise should be about an order of magnitude lower than tf noise.