<p>I would like to use a language that I am familiar with - Java, C#, Ruby, PHP, C/C++, although examples in any language or pseudocode are more than welcome.</p>

<p>What is the best way of splitting a large XML document into smaller sections that are still valid XML? For my purposes, I need to split them into roughly thirds or fourths, but for the sake of providing examples, splitting them into n components would be good.</p>

[{'answer_id': 7853, 'author': 'DannySmurf', 'author_id': 1028, 'author_profile': 'https://Stackoverflow.com/users/1028', 'pm_score': 3, 'selected': True, 'text': '<p>Well of course you can always extract the top-level elements (whether this is the granularity you want is up to you). In C#, you\'d use the XmlDocument class. For example, if your XML file looked something like this:</p>\n\n<pre><code>&lt;Document&gt;\n  &lt;Piece&gt;\n     Some text\n  &lt;/Piece&gt;\n  &lt;Piece&gt;\n     Some other text\n  &lt;/Piece&gt;\n&lt;/Document&gt;\n</code></pre>\n\n<p>then you\'d use code like this to extract all of the Pieces:</p>\n\n<pre><code>XmlDocument doc = new XmlDocument();\ndoc.Load("&lt;path to xml file&gt;");\nXmlNodeList nl = doc.GetElementsByTagName("Piece");\nforeach (XmlNode n in nl)\n{\n    // Do something with each Piece node\n}\n</code></pre>\n\n<p>Once you\'ve got the nodes, you can do something with them in your code, or you can transfer the entire text of the node to its own XML document and act on that as if it were an independent piece of XML (including saving it back to disk, etc).</p>\n'}, {'answer_id': 7861, 'author': 'Rob Cooper', 'author_id': 832, 'author_profile': 'https://Stackoverflow.com/users/832', 'pm_score': 1, 'selected': False, 'text': '<p>This is more of a comment than an answer, but wouldn\'t:</p>\n\n<pre><code>XmlDocument doc = new XmlDocument();\ndoc.Load("path");\n</code></pre>\n\n<p>Read the entire file at once? Just thought I should raise the point since from the look of Thomas\' question, he is concerned about reading large files and wants to break the process down..</p>\n'}, {'answer_id': 7866, 'author': 'Lars Mæhlum', 'author_id': 960, 'author_profile': 'https://Stackoverflow.com/users/960', 'pm_score': 2, 'selected': False, 'text': '<p>As DannySmurf touches on here, it is all about the structure of the xml document.<br />\nIf you only two huge "top level" tags, it will be extremely hard to be able to split it in a way that makes it possible to both merge it back together and read it piece by piece as valid xml.\n<br />\n<br />\nGiven a document with a lot of seperate pieces like the ones in DannySmurfs example, it should be fairly easy.<br />\nSome rough code in Pseudo C# :</p>\n\n<pre><code>int nrOfPieces = 5;\nXmlDocument xmlOriginal = some input parameter..\n\n// construct the list we need, and fill it with XmlDocuments..\nvar xmlList = new List&lt;XmlDocument&gt;();\nfor (int i = 0; i &lt; nrOfPieces ; i++)\n{\n    var xmlDoc = new XmlDocument();\n    xmlDoc.ChildNodes.Add(new XmlNode(xmlOriginal.FistNode.Name));\n    xmlList.Add(xmlDoc);\n}\n\nvar nodeList = xmlOriginal.GetElementsByTagName("Piece")M\n// Copy the nodes from the original into the pieces..\nfor (int i = 0; i &lt; nodeList .Count; i++)\n{\n    var xmlDoc = xmlList[i % nrOfPieces];\n    var nodeToCopy = nodeList[i].Clone();\n    xmlDoc.FirstNode.ChildNodes.Add(nodeToCopy);\n}\n</code></pre>\n\n<p>This should give you n docs with correct xml and the possibility to merge them back together.<br />\nBut again, it depends on the xml file.</p>\n'}, {'answer_id': 7869, 'author': 'DannySmurf', 'author_id': 1028, 'author_profile': 'https://Stackoverflow.com/users/1028', 'pm_score': 1, 'selected': False, 'text': '<p>It would read the entire file at once. In my experience, though, if you\'re just reading the file, doing some processing (i.e., breaking it up) and then continuing on with your work, the XmlDocument is going to go through it\'s create/read/collect cycle so quickly that it likely won\'t matter.</p>\n\n<p>Of course, that depends on what a "large" file is. If it\'s a 30 MB XML file (which I would consider large for an XML file), it probably won\'t make any difference. If it\'s a 500 MB XML file, using XmlDocument will become extremely problematic on systems without a significant amount of RAM (in that case, however, I\'d argue that the time to manually pick through the file with a XmlReader would be the more significant impediment).</p>\n'}, {'answer_id': 72035, 'author': 'Dean Poulin', 'author_id': 5462, 'author_profile': 'https://Stackoverflow.com/users/5462', 'pm_score': 0, 'selected': False, 'text': '<p>It looks like you\'re working with C# and .NET 3.5.  I have come across some posts that suggest using a yield type of algorithm on a file stream with an XmlReader.</p>\n\n<p>Here\'s a couple blog posts to get you started down the path:</p>\n\n<ul>\n<li><a href="http://blogs.msdn.com/xmlteam/archive/2007/03/05/streaming-with-linq-to-xml-part-1.aspx" rel="nofollow noreferrer">Streaming With Linq to SQL Part 1</a> </li>\n<li><a href="http://blogs.msdn.com/xmlteam/archive/2007/03/24/streaming-with-linq-to-xml-part-2.aspx" rel="nofollow noreferrer">Streaming With Linq To SQL Part 2</a></li>\n</ul>\n'}, {'answer_id': 72111, 'author': 'Community', 'author_id': -1, 'author_profile': 'https://Stackoverflow.com/users/-1', 'pm_score': 0, 'selected': False, 'text': '<p>Not sure what type of processing you\'re doing, but for very large XML, I\'ve always been a fan of event-based processing. Maybe it\'s my Java background, but I really do like SAX. You need to do your own state management, but once you get past that, it\'s a very efficient method of parsing XML.</p>\n\n<p><a href="http://saxdotnet.sourceforge.net/" rel="nofollow noreferrer">http://saxdotnet.sourceforge.net/</a></p>\n'}, {'answer_id': 72189, 'author': 'Community', 'author_id': -1, 'author_profile': 'https://Stackoverflow.com/users/-1', 'pm_score': 0, 'selected': False, 'text': '<p>I\'m going to go with youphoric on this one. For very large files SAX (or any other streaming parser) will be a great help in processing. Using DOM you can collect just top level nodes, but you still have to parse the entire document to do it...using a streaming parser and event-based processing lets you "skip" the nodes you aren\'t interested in; makes the processing faster. </p>\n'}, {'answer_id': 73727, 'author': 'mirod', 'author_id': 11095, 'author_profile': 'https://Stackoverflow.com/users/11095', 'pm_score': 0, 'selected': False, 'text': '<p>If you are not completely allergic to Perl, then <a href="http://search.cpan.org/dist/XML-Twig/" rel="nofollow noreferrer" title="CPAN page for the module">XML::Twig</a> comes with a tool named  <a href="http://search.cpan.org/dist/XML-Twig/tools/xml_split/xml_split" rel="nofollow noreferrer" title="xml_split documentation">xml_split</a> that can split a document, producing well-formed XML section. You can split on a level of the tree, by size or on an XPath expression.</p>\n'}, {'answer_id': 313067, 'author': 'Jonas Engman', 'author_id': 4164, 'author_profile': 'https://Stackoverflow.com/users/4164', 'pm_score': 3, 'selected': False, 'text': '<p>Parsing XML documents using DOM doesn\'t scale.</p>\n\n<p>This <a href="http://groovy.codehaus.org/" rel="nofollow noreferrer">Groovy</a>-script is using StAX (Streaming API for XML) to split an XML document between the top-level elements (that shares the same QName as the first child of the root-document). It\'s pretty fast, handles arbitrary large documents and is very useful when you want to split a large batch-file into smaller pieces.</p>\n\n<p>Requires Groovy on Java 6 or a StAX API and implementation such as <a href="http://woodstox.codehaus.org/" rel="nofollow noreferrer">Woodstox</a> in the CLASSPATH</p>\n\n<pre><code>import javax.xml.stream.*\n\npieces = 5\ninput = "input.xml"\noutput = "output_%04d.xml"\neventFactory = XMLEventFactory.newInstance()\nfileNumber = elementCount = 0\n\ndef createEventReader() {\n    reader = XMLInputFactory.newInstance().createXMLEventReader(new FileInputStream(input))\n    start = reader.next()\n    root = reader.nextTag()\n    firstChild = reader.nextTag()\n    return reader\n}\n\ndef createNextEventWriter () {\n    println "Writing to \'${filename = String.format(output, ++fileNumber)}\'"\n    writer = XMLOutputFactory.newInstance().createXMLEventWriter(new FileOutputStream(filename), start.characterEncodingScheme)\n    writer.add(start)\n    writer.add(root)\n    return writer\n}\n\nelements = createEventReader().findAll { it.startElement &amp;&amp; it.name == firstChild.name }.size()\nprintln "Splitting ${elements} &lt;${firstChild.name.localPart}&gt; elements into ${pieces} pieces"\nchunkSize = elements / pieces\nwriter = createNextEventWriter()\nwriter.add(firstChild)\ncreateEventReader().each { \n    if (it.startElement &amp;&amp; it.name == firstChild.name) {\n        if (++elementCount &gt; chunkSize) {\n            writer.add(eventFactory.createEndDocument())\n            writer.flush()\n            writer = createNextEventWriter()\n            elementCount = 0\n        }\n    }\n    writer.add(it)\n}\nwriter.flush()\n</code></pre>\n'}, {'answer_id': 1018858, 'author': 'Ben Bryant', 'author_id': 28953, 'author_profile': 'https://Stackoverflow.com/users/28953', 'pm_score': 0, 'selected': False, 'text': '<p>I did a YouTube video showing <a href="http://www.youtube.com/watch?v=9ANBa9i5LhM" rel="nofollow noreferrer">how to split XML files</a> with <a href="http://www.firstobject.com/dn_editor.htm" rel="nofollow noreferrer">foxe</a> (the free XML editor from <a href="http://www.firstobject.com/about.htm" rel="nofollow noreferrer">Firstobject</a>) using only a small amount of memory regardless of the size of the input and output files.</p>\n\n<p>The memory usage for this CMarkup XML reader (pull parser) and XML writer solution depends on the size of the subdocuments that are individually transferred from the input file to the output files, or the minimum block size of 16 KB.</p>\n\n<pre>split()\n{\n  CMarkup xmlInput, xmlOutput;\n  xmlInput.Open( "50MB.xml", MDF_READFILE );\n  int nObjectCount = 0, nFileCount = 0;\n  while ( xmlInput.FindElem("//ACT") )\n  {\n    if ( nObjectCount == 0 )\n    {\n      ++nFileCount;\n      xmlOutput.Open( "piece" + nFileCount + ".xml", MDF_WRITEFILE );\n      xmlOutput.AddElem( "root" );\n      xmlOutput.IntoElem();\n    }\n    xmlOutput.AddSubDoc( xmlInput.GetSubDoc() );\n    ++nObjectCount;\n    if ( nObjectCount == 5 )\n    {\n      xmlOutput.Close();\n      nObjectCount = 0;\n    }\n  }\n  if ( nObjectCount )\n    xmlOutput.Close();\n  xmlInput.Close();\n  return nFileCount;\n}</pre>\n'}]
['https://Stackoverflow.com/questions/7846', 'https://Stackoverflow.com', 'https://Stackoverflow.com/users/572/']