Thank you both for your help. derHugo is right, my SetObjectVisibility(false); was executed only on server. I changed my HideObject method so it calls ClientRpc method for setting object visibility and now it works.

[Server]
public void HideObject(NetworkConnectionToClient client)
{
    RpcSetObjectVisibility(false);
    TargetSetActive(client);
}

[ClientRpc]
private void RpcSetObjectVisibility(bool visible)
{
    SetObjectVisibility(visible);
}