That happens because electric-indent-mode affects the behavior of comment-line when in org-mode, and possibly other modes as well.

To fix it you can disable electric-indent-mode entirely.

(electric-indent-mode -1)