It seems it is hardcoded in .NET Framework. See the following code: https://referencesource.microsoft.com/#System.Runtime.Serialization/System/Xml/XmlMtomReader.cs,1101
It is using 4096 bytes max for MTOM with XOP. Binary Encoding works fine, but is proprietary.