I don't quite understand "the distance between a quadratic function and R".

If you are looking for the distance between a point and an n-sphere centered at c (i.e. $$(x - c)^{T}(x - c) \le r^{2})$$) you will want the convex program $$\min_x |x - p| \quad \text{subject to} \quad (x - c)^{T}(x - c) \le r^{2}$$

Notice that this cost is not going to be a polynomial since it needs to be 0 on the entire inside of the n-sphere.

As for the claim that $$|p^{T}p + b^{T}p + c - R|_{2}$$ is convex — this is not true in general. Draw the plot of $$|p^{2} - 1|$$ and you'll see the non-convexity.