This is a complete guide to using HXT for parsing and processing HTML in Haskell.
HXT is a collection of tools for processing XML with Haskell. It's a complex beast, but HXT is powerful and flexible, and very elegant once you know how to use it.
TagSoup is the crowd favorite for HTML scraping in Haskell, but it's a bit too basic for my needs.
HXT is based on HaXml. The two are very similar, but I think HXT is a little more elegant.
hexpat is a high-performance xml parser. It might be more appropriate depending on your use case. hexpat lacks a collection of tools for processing the HTML, but you can try Parsec for that bit.
I haven't used Text.XML.Light. If you have used it and liked it, please let me know!
The one thing all these packages have in common is poor documentation.
links <- runX $ doc //> hasName "a" >>> getAttrValue "href"
I don't assume any prior knowledge of Arrows. In fact, one of the goals of this guide is to help you understand Arrows a little better.
Arrows sound like functions! In fact, functions are arrows.
Don't get confused by the two different type signatures! Int -> Bool is just the infix way of writing (->) Int Bool.
You'll be using >>> a lot with HXT, so it's a good idea to understand how it works.
>>> composes two arrows into a new arrow.
We could compose length and odd like so: odd . length.
Since functions are Arrows, we could also compose them like so: length >>> odd or odd <<< length.
They're all exactly the same!
A function is the most basic type of arrow, but there are many other types. HXT defines its own Arrows, and we will be working with them a lot.
Let's get started. Don't worry if Arrows still seem unclear. We will be writing a lot of examples, so they should become clear soon enough.
HandsomeSoup contains a powerful css function that will allow us to access elements using css selectors. We will use this function until we can write a basic version of it ourselves as explained here. For more info about HandsomeSoup, see this section.
doc is now a parsed HTML document, ready to be processed!
ghci> doc >>> css "a"
So css takes an XmlTree and returns another XmlTree. A lot of Arrows in HXT have this type: they all transform the current tree and return a new tree.
Use runX to extract the contents.
Note: To keep things simple for now, these examples make use of our custom css Arrow.
doc >>> css "a" >>> hasAttr "id"
doc >>> css "a" >>> getAttrValue "href"
ghci>runX $ doc >>> css "a" >>> getAttrValue "href"
This is a core idea behind HXT. In HXT, everything you do is a series of transformations on the whole tree. So you can use getAttrValue and HXT will automatically apply it to all the elements.
doc >>> css "p" <+> css "a"
We used the special function "//>" here! It's covered in this section.
Get all elements where the text contains "mouse"
Let's talk about the &&& function.
&&& takes two arrows and creates a new arrow. In the above example, the output of length is fed into both odd and (+1), and both return values are combined into a tuple (True, 4).
We used &&& to get an element's name and its id: (getElemName &&& getAttrValue "id").
HXT has lots of other arrows for selecting elements. See the docs for more.
HXT has a few different functions for working with children, and it can be tricky to decide which one to use.
css uses hasName to get elements with a given tag. Why don't we just use hasName instead of css?
ghci> runX $ doc >>> hasName "a"
multi recursively applies an Arrow to an entire subtree. css uses multi to search across the entire tree for nodes.
These two Arrows are related to multi.
-- so it didn't go through the child nodes of that element.
/> looks for a direct child (i.e. what getChildren does).
//> looks for a node somewhere under this one (i.e. what deep does).
Try using /> instead of //>. What do you get?
This returns  because doc >>> css "a" >>> getText returns .
Use removeAllWhiteSpace. It removes all nodes containing only whitespace.
If you have used BeautifulSoup, this is kinda like the stripped_strings method.
runX . xshow $ doc >>> css "p" >>> addAttr "id" "my-own-id"
processChildren and processTopDown allow you to modify the children of an element.
processChildren is similar to getChildren, except that instead of returning the children, it modifies them in place and returns the entire tree.
processTopDown is similar to multi.
processTopDownUntil is similar to deep.
HXT has some useful functions that allow us to apply Arrows based on a predicate.
ifA is the if statement for Arrows. It's used as ifA (predicate Arrow) (do if true) (do if false).
We use the identity arrow this here. You can read this as: if the element is a p tag, uppercase it, otherwise pass it through unchanged.
when and guards can make your ifA code easier to read.
f `when` g -- when the predicate `g` holds, `f` is applied, else the identity filter `this`.
g `guards` f -- when the predicate `g` holds, `f` is applied, else `none`.
First, import Text.XML.HXT.DOM.XmlNode. It defines several functions that work on Nodes.
(Note the qualified import...this module has a lot of names that conflict with HXT.Core).
We can use isA wherever a predicate Arrow is needed: ifA, when, guards etc.
See the docs for more conditionals for Arrows.
See these docs for more functions you can use to write your own Arrows.
And we want to get the length of each bit of text. So we need an arrow version of the length function.
Note how length automatically gets applied to each element without us having to use map. This is because Arrows in HXT always apply to the entire tree, not just one node. This behaviour is abstracted away so that you can just write a function that works on one node and have it apply to every node in the tree automatically.
This section was written after Ywen asked this question on Reddit. So far, we have applied arrows to one node at a time. In the previous section, we applied length to every node individually. What if we wanted to work with all the nodes at once, to do a map or a foldl over them?
HXT has some special functions that allow you to work on the entire list of elements, instead of working on just one element.
>>. takes a function that takes a list, and returns a list, so it allows us to use all our Haskell list functions.
>. takes a function that takes a list and returns a single value.
The parentheses are important here!
doc >>> css "li > a.link:first-child"
doc >>> css "a" ! "href"
links <- runX $ doc >>> css "a" ! "href"
It's in IO! This means that any function that parses the html will have to be IO. What if you want a pure function for parsing the html?
old :: IO [XmlTree] -- IO!
new :: [XmlTree] -- no IO!
parse a string as HTML content, substitute all HTML entity refs and canonicalize tree. (substitute char refs, ...). Errors are ignored. This is a simpler version of readFromString without any options.
HXT provides arrows to print out the current tree at any time. These arrows are very handy for debugging.
doc >>> css "h1" >>> withTraceLevel 5 traceTree >>> getAttrValue "id"
traceTree needs level >= 4.
doc >>> css "h1" >>> traceMsg 1 "got h1 elements" >>> getAttrValue "id"
See the docs for even more trace functions.
I hope you found this guide helpful in your quest to work with HTML using Haskell.
Arrows for working with nodes (the core stuff).
Arrows for working with children.
Function versions of most Arrows (Useful with arr or isA).
The HXT tutorial on haskell.org.