   [.net] Two problems with System.Xml
I'm having a few problems with System.Xml. My program is an English-Hungarian dictionary, and its input files are in the TEI XML format.Here's an example of one of my TEI files. <?xml version="1.0" encoding="utf-8"?><?xml version="1.0" encoding="utf-8"?><!DOCTYPE  TEI.2  PUBLIC "-//TEI Consortium//DTD TEI P4//EN"  "http://www.tei-c.org/P4X/DTD/tei2.dtd" [  <!ENTITY % TEI.XML"INCLUDE" >  <!ENTITY % TEI.dictionaries "INCLUDE" >  <!ENTITY % TEI.linking    "INCLUDE" >  <!ATTLIST xptr url CDATA #IMPLIED >  <!ATTLIST xref url CDATA #IMPLIED >]><TEI.2>  <teiHeader>    <fileDesc> <titleStmt>   <title>A very sternly abridged English-Hungarian Dictionary</title>   <respStmt><resp>created by</resp><name>Lee Houghton</name>   </respStmt> </titleStmt> <editionStmt>   <edition>0.0.1</edition> </editionStmt> <extent>as I said, it's very sternly abridged</extent> <publicationStmt>   <publisher>Lee Houghton</publisher>   <availability><p>  Available under the terms of the DWTFYWT  Public License, available at <xptr url="http://sam.zoy.org/wtfpl/"/>.</p>   </availability>   <date>2007-09-03</date>   <pubPlace>http://TODO/</pubPlace> </publicationStmt> <notesStmt>   <note type="status">testing</note>   <!-- descriptive adjective, eg. "stable" --> </notesStmt> <sourceDesc>   <p>Home: <xptr url="http://asztal.net/szotar/"/>   </p>   <p>This file was generated by hand.</p> </sourceDesc>    </fileDesc>    <encodingDesc> <projectDesc>   <p>This dictionary is part of the Szótár project aimed at providing...well, something, anyway.   </p> </projectDesc>    </encodingDesc>  </teiHeader>  <text>    <body> <div0 type="dictionary" n="English-Hungarian">   <div1 type="chapter" n="t"><entry>  <form>    <orth>table</orth>  </form>  <sense n="1">    <gramGrp> <pos>n</pos>    </gramGrp>    <def>an article of furniture consisting of a flat, slab-like top supported on one or more legs or other supports.</def>    <eg> <q>a kitchen table</q>    </eg>    <trans> <tr>asztal</tr>    </trans>  </sense>  <sense n="2">    <def>an arrangement of words, numbers, or signs, or combinations of them, as in parallel columns, to exhibit a set of facts or relations in a definite, compact, and comprehensive form; a synopsis or scheme.</def>    <gramGrp> <pos>n</pos>    </gramGrp>    <trans> <tr>táblázat</tr>    </trans>  </sense>  <sense n="3">    <gramGrp> <pos>n</pos> <number>pl</number>    </gramGrp>    <def>the people sitting at a table</def>    <trans> <tr>asztaltársaság</tr>    </trans>  </sense></entry>   </div1> </div0> <div0 type="dictionary" n="Hungarian-English">   <div1 type="chapter" n="a"><entry>  <form>    <orth>asztal</orth>  </form>  <sense n="1">    <trans> <tr>table</tr>    </trans>  </sense>  <sense n="2">    <usg type="dom">comp</usg>    <trans> <tr>desktop</tr>    </trans>  </sense></entry>   </div1> </div0>    </body>  </text></TEI.2>Problem 1: It marks some valid DTDs as invalidI would like to have my input files validated, but the TEI DTD is quite complex. Apparently too complex for XmlTextReader, which thinks that the DTD is invalid. I've run the document through some other validators which seem to agree that it's valid - and at one point I had the IDE complaining that part of a comment inside the DTD was invalid. So far I've just hacked around it by writing an XmlResolver that simply returns an empty MemoryStream when the reader requests the DTD (or indeed any file), which has allowed me to actually load the file, but it's a bad solution, as things such as external entity declarations won't load.I've found no way to get XmlReader to ignore the DTD completely (I can prohibit the DTD, but that's not useful to me). Do I just give up with DTDs? Perhaps try to create an XSD schema of my own?Problem 2: Loading fragments of the documentThe files I have at the moment are about 25MB in size, which causes some problems simply loading in an XmlDocument. It takes about 6 seconds to call XmlDocument.Load on my test file, whereas it took about 2.5 seconds and a tenth of the memory to actually do work on it with XmlReader.Initially only parts of the entry are loaded, the orth and tr, which are used to create a summary used for searching. I'd like to load the actual entry data only when it's needed, so I think I'll need to store the position of each entry. Only I don't seem to be able to get at that data in any nice way.XmlTextReader won't actually tell me what byte offset it is at in the stream, and the underlying stream position doesn't point to it. The only hint I'm given is the line number and column, which I really don't want to have to use.tl;dr: two main questions - how can I get XmlTextReader to ingore my DTD and how can I load fragments of the XML document only when they are needed?Edit: I guess I'll just go with creating an index file for #2 and ignoring the DTDs as I currently do for #1. The index file was probably necessary to load within less than a second anyway, but I'm lazy [sad]However, finding positions in the file by line numbers seems very flaky.  Currently I'm opening the stream with an XmlReader to find the encoding then seeking back to the start and reading each line with a StreamReader with that encoding. It seems like it could fail if the encoding of the XML declaration itself (I think this is either always utf-8, or determined by the byte order mark) is not the same as the encoding of the data following it. Does XML allow this?[Edited by - MrEvil on September 10, 2007 6:23:30 AM]