The problem comes from this source file. The function is php_sockop_close and probably the problem is still active in the most recent php version. I think it should check for whether the socket being closed is a server (listening) or a client socket and do what it does with the 500ms timeout on a client socket only.
The workarounds:
patch and recompile;
patch (hex edit) the php5ts.dll and set whatever value might suit you;
rework the server so you use socket_* functions and not stream_socket_* ones;
Unfortunately turning a stream into a socket using socket_import_stream and using socket_close will not work - the same timeout will occur;
And unfortunately socket_export_stream is not available for PHP 5.4.