To answer your last statement, yes I would be concerned with a loss of data due to removing a participant if you're trying to analyze the whole population with one polychoric correlation test.

I would split your population into subgroups based on their answer to a conditional question (i.e. all participants that answered yes to "Do you eat ice cream?". You would then perform the polychoric correlation on each of those subgroups.