This issue is likely related to MSDTC session timeouts and the way DTC handles idle connections in unauthenticated, cross-domain scenarios. Since you've already confirmed that:
You’re using "No Authentication Required" mode,
The DTC handshake completes successfully on the second try (within a 10-minute window),
And the issue is repeatable after a period of inactivity
…it suggests that the DTC session is being closed due to idle timeout, and the first transaction after that fails due to a cold handshake or unavailable session cache.
Explanation
MSDTC uses a combination of session-level security and RPC-based communication, which can be sensitive to:
Network security policies (e.g., firewalls or timeouts on idle RPC sessions),
Authentication settings (especially in cross-domain, unauthenticated environments),
DTC session cache expiration.
In environments where No Authentication Required is set, MSDTC skips mutual authentication and relies more heavily on initial handshakes. When idle, the DTC service may discard session-related state, leading to the need for a full handshake again — which sometimes fails due to timing, firewall rules, or race conditions.