This is what really worked for me

add_action('pre_get_comments', function($query) {
    if ( !is_admin()) {
        $query->query_vars['order'] = 'DESC';
    }
});

Intercepting comments_template_query_args still did not work as pagination sorting is messed.