title: Heading identification with Regex
body: <p>I'm wondering how I can identify headings with differing numerical marking styles with one or more regular expressions assuming sometimes styles overlap between documents. The goal is to extract all the subheadings and data for a specific heading in each file, but these files aren't standardized. Is regular expressions even the right approach here?</p>
  
  <p>I'm working on a program that parses a .pdf file and looks for a specific section. Once it finds the section it finds all subsections of that section and their content and stores it in a <code>dictionary&lt;string, string&gt;</code>. I start by reading the entire pdf into a string, and then use this function to locate the "marking" section.</p>
  
  <pre><code>private string GetMarkingSection(string text)
      {
        int startIndex = 0;
        int endIndex = 0;
        bool startIndexFound = false;
        Regex rx = new Regex(HEADINGREGEX);
        foreach (Match match in rx.Matches(text))
        {
          if (startIndexFound)
          {
            endIndex = match.Index;
            break;
          }
          if (match.ToString().ToLower().Contains("marking"))
          {
            startIndex = match.Index;
            startIndexFound = true;
          }
        }
        return text.Substring(startIndex, (endIndex - startIndex));
      }
  </code></pre>
  
  <p>Once the marking section is found, I use this to find subsections.</p>
  
  <pre><code>private Dictionary&lt;string, string&gt; GetSubsections(string text)
      {
        Dictionary&lt;string, string&gt; subsections = new Dictionary&lt;string, string&gt;();
        string[] unprocessedSubSecs = Regex.Split(text, SUBSECTIONREGEX);
        string title = "";
        string content = "";
        foreach(string s in unprocessedSubSecs)
        {
          if(s != "") //sometimes it pulls in empty strings
          {
            Match m = Regex.Match(s, SUBSECTIONREGEX);
            if (m.Success)
            {
              title = s;
            }
            else
            {
              content = s;
              if (!String.IsNullOrWhiteSpace(content) &amp;&amp; !String.IsNullOrWhiteSpace(title))
              {
                subsections.Add(title, content);
              }
            }
          }
        }
        return subsections;
      }
  </code></pre>
  
  <p>Getting these methods to work the way I want them to isn't an issue, the problem is getting them to work with each of the documents. <strong>I'm working on a commercial application so any API that requires a license isn't going to work for me.</strong>
  These documents are anywhere from 1-16 years old, so the formatting varies quite a bit. <a href="https://regex101.com/r/6RckWL/5" rel="nofollow noreferrer">Here is a link to some sample headings and subheadings from various documents.</a> But to make it easy, here are the regex patterns I'm using:</p>
  
  <ul>
  <li>Heading: <code>(?m)^(\d+\.\d+\s[ \w,\-]+)\r?$</code></li>
  <li>Subheading: <code>(?m)^(\d\.[\d.]+ ?[ \w]+) ?\r?$</code></li>
  <li>Master Key: <code>(?m)^(\d\.?[\d.]*? ?[ \-,:\w]+) ?\r?$</code></li>
  </ul>
  
  <p>Since some headings use the subheading format in other documents I am unable to use the same heading regex for each file, and the same goes for my subheading regex.</p>
  
  <p>My alternative to this was that I was going to write a master key (listed in the regex link) to identify all types of headings and then locate the last instance of a numeric character in each heading (5.1.X) and then look for 5.1.X+1 to find the end of that section.</p>
  
  <p>That's when I ran into another problem. Some of these files have absolutely no proper structure. Most of them go from 5.2->7.1.5 (5.2->5.3/6.0 would be expected)</p>
  
  <p>I'm trying to wrap my head around a solution for something like this, but I've got nothing... I am open to ideas not involving regex as well.</p>
  
  <p>Here is my updated <code>GetMarkingSection</code> method:</p>
  
  <pre><code>private Dictionary&lt;string, string&gt; GetMarkingSection(string text)
      {
        var headingRegex = HEADING1REGEX;
        var subheadingRegex = HEADING2REGEX;
        Dictionary&lt;string, string&gt; markingSection = new Dictionary&lt;string, string&gt;();
  
        if (Regex.Matches(text, HEADING1REGEX, RegexOptions.Multiline | RegexOptions.Singleline).Count &gt; 0)
        {
          foreach (Match m in Regex.Matches(text, headingRegex, RegexOptions.Multiline | RegexOptions.Singleline))
          {
            if (Regex.IsMatch(m.ToString(), HEADINGMASTERKEY))
            {
              if (m.Groups[2].Value.ToLower().Contains("marking"))
              {
                var subheadings = Regex.Matches(m.ToString(), subheadingRegex, RegexOptions.Multiline | RegexOptions.Singleline);
                foreach (Match s in subheadings)
                {
                  markingSection.Add(s.Groups[1].Value + " " + s.Groups[2].Value, s.Groups[3].Value);
                }
                return markingSection;
              }
            }
          }
        }
        else
        {
          headingRegex = HEADING2REGEX;
          subheadingRegex = HEADING3REGEX;
  
          foreach(Match m in Regex.Matches(text, headingRegex, RegexOptions.Multiline | RegexOptions.Singleline))
          {
            if(Regex.IsMatch(m.ToString(), HEADINGMASTERKEY))
            {
              if (m.Groups[2].Value.ToLower().Contains("marking"))
              {
                var subheadings = Regex.Matches(m.ToString(), subheadingRegex, RegexOptions.Multiline | RegexOptions.Singleline);
                foreach (Match s in subheadings)
                {
                  markingSection.Add(s.Groups[1].Value + " " + s.Groups[2].Value, s.Groups[3].Value);
                }
                return markingSection;
              }
            }
          }
        }
        return null;
      }
  </code></pre>
  
  <p>Here are some example PDF files:
  <a href="https://i.stack.imgur.com/ifAsz.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/ifAsz.png" alt="Style 1"></a>
  <a href="https://i.stack.imgur.com/z1ApI.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/z1ApI.png" alt="Style 2"></a></p>