Not sure it would cover all edge cases but using CSS.escape on an invalid selector returns an empty NodeList instead of throwing a syntax error.

document.querySelectorAll(CSS.escape('p > > > a')) => NodeList []
document.querySelectorAll(CSS.escape('a?+')) => NodeList []