I wanted to share my observations with "ODBC Driver 18 for SQL Server".

Doing SQLSetConnectAttr(..., SQL_ATTR_ENLIST_IN_XA, OP_START, ...) and then SQLSetConnectAttr(..., SQL_ATTR_TXN_ISOLATION, ...), gives me the error in SQLGetDiagRec() "Operation invalid at this time".

Doing your suggested XID Data layout, didn't work for version 18.
I get XA Error (-8) (XAER_DUPID, Duplicate XID) whenever i do gtrid (length 24) at data[0] and bqual (length 12) at data[64].
My guess is, that MSDTC will read this like other DB's now:
gtrid = data[0] and bqual = data[24]
bqual will be always the same (0') => Duplicate XID.

"The incoming tabular data stream (TDS) protocol stream is incorrect. The stream ended unexpectedly."
If someone has this error at SQLGetDiagRec(), make sure you have SQL_AUTOCOMMIT_ON when enlisting XID's and SQL_AUTOCOMMIT_OFF when done enlisting.
This has costed me a lot of time to figure out.

Besides executing the Query from @nfrmtkr above, I would suggest setting TRACE_XA to 4 (Verbose) and the TraceFilePath.
You can see each XA Call with a few details in the log file.
For Windows: https://learn.microsoft.com/en-us/troubleshoot/windows/win32/enable-diagnostic-tracing-ms-dtc
On Linux you can use the tool mssql-conf.
/opt/mssql/bin/mssql-conf set distributedtransaction.trace_xa 4
/opt/mssql/bin/mssql-conf set distributedtransaction.tracefilepath /tmp
Restart SQL Server afterwards.
The file will look like this /tmp/MSDTC-sqlservr.exe-444.log