First of all,

SMB connector is not reliable connector. Alternative you should use JAVA library like smbj which is SMB client Implementation.

Now coming to your issues: Try below

• Server is closing connections due to too many open connections from the client as SMB connector does not support connection Pooling which might be one of reason. So just to validate my answer or fix the issues, Capture the TCPDump or Packet traffic from MuleSoft server and check why RST flag coming from server, which would tell you everything. Trust me.
• Work with Network Team and server team to fix such issues.
• As I do not have access to log , do one thing Wrap SMB operations in a try-catch block.There might be case that reconnecton is not coming into effect.
• You should set socket timeout like 10 minutes or 20 minutes ,because If server has shorter Ideal timeout it would reset the connection and your application acting as a client would be in trouble because of 10 mintutes.