Glad you found your error.

One method I always use is to set alpha to a very small value. For a tiny alpha, it should converge. If it doesn't, you have an error in your code.

Draw a graph. if it's not plateauing, check your code. Make sure you subtract the alpha * gradient, not add it - common mistake.