In the end I just decided to go with the easiest and for me most flexible option by just ignoring the namespaces in the XML.

public class IgnoreNamespaceXmlTextReader : XmlTextReader
{
    public IgnoreNamespaceXmlTextReader(TextReader reader) : base(reader) { }

    public override string NamespaceURI => string.Empty; // Ignores all namespaces
}

Not saying its the best solution, but for my use I think I can afford to do this.