First, use the "editor.colorDecorators": false to disable the feature for all languages. Then, use language-specific settings to re-enable it only for CSS.
In settings.json:
{
"editor.colorDecorators": false,
"[css]": {
"editor.colorDecorators": true
}
}