I realized that because the gradient of this function (with respect to w) being 2*x*cos(w*x)*sin(w*x), it depends linearly on x. And since I am using lots of large input combined with small input, the gradient is essentially random.

I was able to successfully train the 'network' using just 6 sample points (0 - 5).

To use all the data, I guess, I will have to come up with a well behaved gradient function which does not depend too much on the variable x.