I was led here by google for a similar problem with connecting plot points. However, this had nothing to do with summary statistics.

My solution was to use geom_path() instead of geom_line().

Apparently it was decided to have the line connect everything "left to right", and the path follows your order of co-ordinates.