Following samcarter_is_at_topanswers.xyz's answer, I looked into the moderncv documentation, and found besides loading \moderncvcolor{} before \moderncvstyle{}, we also need to use \colorlet{lastnamecolor}{black} and \colorlet{firstnamecolor}{black} (called after \moderncvstyle{}) to make the color of name looks right.