So this doesn't work for me as well

tomSelect.settings.plugins = ["remove_button"];

So i've found this in the doc and it works for me.

https://tom-select.js.org/plugins/remove-button/#remove-button

new TomSelect('#input-tags',{
    plugins: {
        remove_button:{
            title:'Remove this item',
        }
    },
    
});