In this article we will develop a set of transducers which can be composed to extract data from an XML document (although the principles here can be applied to any tree-structured format, including JSON).
The built-in Clojure library already has support for zippers to navigate an XML document. But there are limitations. Zippers are complicated animals to work with, take quite a lot of setting up and require that the document fits in memory. As an experiment, let's see how far we can get with transducers instead.
Let's start with an example XML file, which we'll save to example.xml.
Clojure already contains an XML parser. We'll use it to parse our document into a Clojure map.
The first transducer we shall write will be one that descends into the node's child elements.
Transducers work on sequences (or streams) of data. Since we want to get the children of multiple nodes into a single combined sequence, we'll use mapcat rather than map.
Give it a sequence of nodes and it will return a sequence of children.
Usually we want to select children by some criteria. It's very common to select child elements by their tag names.
So let's compose a transducer that will filter children by their tag name, using an arbitary predicate.
We've just created a function, named tagp, that given a predicate, returns a transducer which selects some nodes' children and filters them by their tag names.
It's very common to want to use an equality predicate, so we can select children by tag name. So let's create a convenience function for this.
Now we can select all the chapter nodes of our example doc like this.
Note that we have only replaced the children transducer with our new 'get elements by tag name' one.
Similarly, we can build attrp and attr= to select children based on their attributes. Since the attributes are contained in the :attrs entry, we need to compose an accessor to access them.
Now we can create our 2 attribute transducers.
Let's use our attr= transducer to count how many chapters have an attribute name of 'Conclusion'.
Let's re-write that code above to expose a path syntax. We simply apply comp over the path to yield the same composite transducer as before.
Note how we have now built a primitive, yet arguably more powerful, version of XPath.
We could implement other XPath node selectors in the same straight-forward way.
We can now compose all these transducers to extract paragraphs in our document. Let's extract just the paragraphs of the chapter named 'Introduction'.
One reason why this approach is more powerful than XPath is that we are not limited to using XPath's built-in functions, but can inject custom transducers or any other functions into our 'path' to do anything we like, even something that's would be impossible in XPath, such as creating a map between the chapter names and the text of their paragraphs.
Nor are we limited to XML. With some easy modifications, we can build a path query for JSON or other formats.
As an exercise left for the reader, can you implement XPath's descendants as a transducer?
Discuss below or join the discussion on Hacker News.