By setting your RGB data to a default/constant value you'll very likely experience a performance drop compared to the "original" solution as your data are less qualitative. And indeed your model will be slower than a model using only XYZ data, as it has more parameters.