By using the FilePond API which the FileUpload field makes use of, I could accomplish something similar to what I wanted:

FileUpload::make('Files')
           ->acceptedFileTypes(self::$filetypes)
           ->disk('local')
           ->moveFiles()
           ->multiple(true)
           ->extraAlpineAttributes([
                 'x-init' => '$el.addEventListener(\'FilePond:processfileprogress\', (e) => { if(e.detail.file.filename === \'test.csv\') e.detail.file.abortProcessing() })'
           ])
           ->afterStateUpdated(function ($state) {
                  foreach ($state as $idx => $file) {
                      $filename = $file->getClientOriginalName();
                      if ($filename == 'test.csv') {
                         $file->delete();
                      }
                  }
           })

With this method I cannot make it look red like the upload failed. It only says "aborted" and stays grey. Though the contrast to the successful green with multiple files should be good enough.

And I still need to use the ->afterStateUpdated() method to handle things server-side and delete the uploaded files.