Here I will suggest two thing as per below :

1. Use using keyword to closed write object before delete.

   File.Copy(filename, filename + ".bak");
   using (TextWriter writer = new StreamWriter(filename))
   {
       writer.Write(content);
   }
   if (File.Exists(filename + ".bak"))
   {
       try
       {
           File.Delete(filename + ".bak");
       }
       catch (IOException ex)
       {
           Debug.WriteLine("Delete failed: " + ex.Message);
           Thread.Sleep(500); // allow file system to catch up
           File.Delete(filename + ".bak"); // retry
       }
   }
   

2. Also I faced issue is file corrupted when copy from source like FTP So please check before delete you just copy file on destination folder and check is corrupted or not and able to open if are able to open same location then try delete operation.

Sure it will work..