text: Roy Osherove
  January 2004
  Summary: Discover how to use the built-in regular expression functionality in the .NET Framework to convert delimited log files into XML for later processing. (12 printed pages)
  Applies to
  Microsoft® ASP.NET
  Microsoft® .NET Framework
  Microsoft® Visual C#®
  Download the source code for this article.
  What You'll Need
  Introduction
  Parsing Text Easily
  Transforming to XML
  A More Generic Approach
  Putting It All Together
  Letting the User Search for Data
  Summary
  Before diving into this article, you'll want to:
  .
  Taking a look at one of the log files, I saw was built of the following columns, delimited by tabs:
  So, I needed to transform this blob of text and lines into something a little more structured. Something we could search through. You're thinking, "Hmm, why not import the file into Microsoft® Access using the Tab-Delimited Wizard?" That solution would be totally okay, if we had one, or just a small bunch of files. The solution here required a little more automation. Plus, had the log files been written in a different format—for example, several lines per log data—we'd be in trouble.
  What we needed here was a raw searchable data format. Something like XML...
  Thinking more about XML, I could see several benefits. Once these files were in XML, we could:
  But how to perform this magical act? What tools are in the Microsoft® .NET Framework that would allow us to:
  If you've worked with regular expressions before, you know that using them is one of the fastest and easiest ways of parsing text. In the .NET Framework, the main class to be used in this area is the System.Text.RegularExpressions.Regex class. One of the most powerful features of this class is the ability to specify, within the search pattern, groups that will allow parsing and retrieval of parts of the text easily. For example:
  Given the date 17/08/1975, and this regular expression: (?<day>\d{1,2})/(?<month>\d{1,2})/(?<year>(?:\d{4}|\d{2}), you can write code to retrieve any part of the text in the date by name, like so:
  (?<day>\d{1,2})/(?<month>\d{1,2})/(?<year>(?:\d{4}|\d{2})));
  }
  The result is: Day:17,Month:08,Year:1975.
  Note If you don't understand the code above, you should refer to the two articles mentioned at the beginning of this article.
  Once you have a bunch of data, like the date example above, you can easily transform it into XML. The XML representation of the date from the previous example could be represented like so:
  <Date>
  <day>17</day>
  <month>08</month>
  <year>1975</year>
  </Date>
  To output this kind of XML before used to be a pretty easy task, yet also pretty error prone. Sure, you could just slap on string upon string of XML content into a memory buffer, but the amount of errors you could get into was horrible. The XMLTextWriter class in the System.XML namespace rids you of a lot of details here, and very conveniently abstracts away all the boilerplate code you need to write, allowing you to concentrate on the content you wish to write in();
  //we write the XML into an in-memory string buffer
  XmlTextWriter writer =
  new XmlTextWriter(new StringWriter(output));
  writer.Formatting=Formatting.Indented;
  //make the XML human-readable
  //write the Start is a standard XML document
  writer.WriteStartDocument();
  //create the openeing nnode(); //not needed here
  writer.WriteEndDocument();
  //Closes any open elements automatically
  writer.Close();
  return output.ToString();
  }
  }
  Here's the output:
  <?xml version="1.0" encoding="utf-8"?>
  <Date>
  <day>17</day>
  <month>08</month>
  <day>1975</day>
  </Date>
  As you can see, it's a very easy job to write XML using this class.
  First you create an in-memory StringBuilder that will house the created XML. Then hand it off to the constructor of a StringWriter, which is used to contruct your XMLTextWriter object. By the way, you could easily pass any System.IO.StreamWriter-derived object, which gives you the flexibility of writing to pretty much anything you want. You can then call the WriteStartDocument method, which creates the <xml version-..> tag at the beginning of the XML text.
  Now open a new element tag, which will contain sub-elements, using the WriteStartElement method. Proceed to write the actual values as sub nodes in the open element using the WriteElementString method, passing in the name of the node, and the value inside it. To finish it all off, you call the WriteEndDocument method, which closes all open elements in the XML. if you want to just close the current "Date" element, you would call the WriteEndElement method, and continue writing more elements.
  Note If you need to write strings containing characters that are illegal in XML as the names of elements, see the XMLConvert class's EncodeName and DecodeName methods. XMLConvert is a great helper, and you'll need it in many situations.
  Actually, if you want to, you can make the writing function much more generic by automatically going through, you need to have an instance of the Regex class to play with. You have to use this same instance to receive the Match object. Then you can use the Regex instance to retrieve the name of a group based on its number:
  reg.GroupNameFromNumber(i)
  This means that for this functionality to work, you can't use the static Match method of the Regex class. This makes things a bit more cumbersome for us. That's why, for the remainder of the code samples, I'll use the earlier version of the code, although it's less generic. You can then implement this method in your programs if you wish to.
  Okay, so macth:
  The class receives a file name to parse. It creates an in-memory XML text writer object and initializes it to the proper settings. It then creates an open Entries element inside it, in to which all the child Entry elements (for each line) will be written to. It then goes through each line in the log file, and uses the Regex.Match method on that line, using a pattern that matches each sub-group I've identified at the beginning of this article.
  If the match was a success, it passes both the XML writer instance and the Match object to a separate method, which writes the group names and values into the XML writer instance. After going through all the lines in the log file, it closes all the elements in the XML file and writes it all to a file named the same as the original log file with the addition of a ".log" at the end.
  If you now open the generated XML log file, you will see the following:
  <?xml version="1.0" encoding="utf-8"?>
  <Entries>
  <Entry>
  <date>25/05/2002</date>
  .
  .
  .
  .
  </Entries>
  Now that you have your data stored as structured XML, you can use it to let the user easily search through it. To do this, you can use a very easy technique already provided by the .NET Framework—use a DataSet object to load your XML data, then "select" data from the Dataset using a filter that can be specified by the user. You can then display the resulting DataRows to the user.
  The DataSet object has a LoadXML method, which allows you to pass it a file name and have it automatically load the data into a table structure inside the dataset. For our purposes, we can send in the file name without any additional parameters. What will be generated inside the Dataset's memory will be a table that contains a collection of DataRows, each one holding a set of columns that corresponds to the set of properties you create in the log file—Date, Time, Hour, Action and so on. Once you have this table in place, you can use the DataTable's Select method to retrieve any number of DataRow objects that matches the filter, you can let the user load any XML file, and filter its contents based on a SQL-compatible filter expression. Anything that can be written after the 'WHERE' clause of a SQL query, can be specified here.
  You receive an array of DataRows, and since you know beforehand the names of the columns for each DataRow (same as the XML elements in your log file), you can just display the values for each column.
  You could just as easily have looped through all the available columns and displayed each one's value to the user, without even knowing what kind of data was inside your DataRow. You could dynamically add columns to your ListView corresponding to the name of each DataColumn in the DataRow, and voila—you have yourself a more generic searching mechanism for practically any simple XML file.
  Parsing log files is easy. Writing XML files is easy. Searching XML files is easy. Therefore, generating XML log files and searching them should be pretty darn easy!. Roy is also the author of the Feedable service and of the free regular expression tool, The Regulator.