My solution has been to have the labels and datasets on their own (as normal JS objects, no Vue refs or anything), and an updatedData method that returns the object { labels: labels, datasets: datasets }
Then in the update, push into labels and datasets.data and call chartData.value = updatedData(labels, datasets)
This way it triggers the update without making a copy or restructuring the entire arrays, and without the maximum call stack exceeded error.
I have yet to test if the shallowRef approach works