If the underlying model structure changes (the modelStructureChanged method is invoked) the following are reset to their default values: Comparators by column, current sort order, and whether each column is sortable. The default sort order is natural (the same as the model), and columns are sortable by default.

From https://docs.oracle.com/javase/8/docs/api/javax/swing/table/TableRowSorter.html

The documentation says that the state of the TableRowSorter would be reset if there is a change in the structure of the model. This means anytime you cause a change in the structure, you must set the comparators again. Things like changing the number of columns would cause a structure change, whereas things like just setting the value of a cell should not cause a structure change.

So the right way seems to be to set the comparators again whenever you cause a structure change. A possibly hackish way to do that would be to override modelStructureChanged() on TableRowSorter to set the comparators again.