title: Problems with strtok()
body: <p>I have been wrestling with this for a while. I know it's a lot of code to look at, but I have no idea where the problem lies and can't seem to narrow it down. I will bounty it.</p>
  
  <p>I wrote this class to parse bbcodes. It uses strtok() primarily, and the class works great unless you put two tags right next to each other, and I can't for the life of me figure out why.</p>
  
  <p>For instance <code>[b] [i]test1[/i] [/b]</code> results in <code>&lt;strong&gt; &lt;em&gt;test1&lt;/em&gt; &lt;/strong&gt;</code>.
  Yet <code>[b][i]test1[/i][/b]</code> results in <code>&lt;strong&gt;i]test1/b]&lt;/strong&gt;</code>.
  The last <code>&lt;/strong&gt;</code> tag is only in there because the parser automatically closes tags it could not find a closing tag for in the string. It somehow misses the <code>[i]</code> and <code>[/b]</code> tags completely.</p>
  
  <p>Here's the class as well as the one subclass it uses for setting up the various bbcodes. The subclass is basically just a data structure with no behaviours.</p>
  
  <pre><code>&lt;?php
      // beware images can contain any url/any get request. beware of csrf
      class Lev_TextProcessor_Extension_BbCode {
  
          protected $elements = array();
          protected $openTags = array();
  
          public function __construct() {
              $this-&gt;elements['b'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;strong&gt;', '&lt;/strong&gt;');
              $this-&gt;elements['i'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;em&gt;', '&lt;/em&gt;');
              $this-&gt;elements['u'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;span style="text-decoration: underline;"&gt;', '&lt;/span&gt;');
              $this-&gt;elements['s'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;span style="text-decoration: line-through;"&gt;', '&lt;/span&gt;');
              $this-&gt;elements['size'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;span style="font-size: ', '&lt;/span&gt;', 'px;"&gt;');
              $this-&gt;elements['color'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;span style="color: ', '&lt;/span&gt;', ';"&gt;');
              $this-&gt;elements['center'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;div style="text-align: center;"&gt;', '&lt;/div&gt;', '', true, true, false);
              $this-&gt;elements['url'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;a href="', '&lt;/a&gt;', '"&gt;');
              $this-&gt;elements['email'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;a href="mailto:', '&lt;/a&gt;', '"&gt;');
              $this-&gt;elements['img'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;img src="', '" alt="" /&gt;', '', false, false, true);
              $this-&gt;elements['youtube'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;object width="400" height="325"&gt;&lt;param name="movie" value="http://www.youtube.com/v/{param}"&gt;&lt;/param&gt;&lt;embed src="http://www.youtube.com/v/', '" type="application/x-shockwave-flash" width="400" height="325"&gt;&lt;/embed&gt;&lt;/object&gt;', '', false, false, false);
              $this-&gt;elements['code'] = new Lev_TextProcessor_Extension_BbCode_Element('&lt;pre&gt;&lt;code&gt;', '&lt;/code&gt;&lt;/pre&gt;', '', true, false, false);
          }
  
          public function processText($input) {
              // pre processing
              $input = htmlspecialchars($input, ENT_NOQUOTES);
              $input = nl2br($input);
              $input = str_replace(array("\n", "\r"), '', $input);
              // start main processing
              $output = '';
              $allow_child_tags = true;
              $allow_child_quotes = true;
  
              $string_segment = strtok($input, '[');
  
              do {
                  // check content for quotes
                  if ($allow_child_quotes === false) {
                      if (strpos($string_segment, '"') === false) {
                          $output .= $string_segment;
                      }
                  } else {
                      // add content to output
                      $output .= $string_segment;
                  }
  
                  $tag_contents = strtok(']');
  
                  if (strpos($tag_contents, '/') === 0) {
                      // closing tag
                      $tag = substr($tag_contents, 1);
                      if (isset($this-&gt;elements[$tag]) === true &amp;&amp; array_search($tag, $this-&gt;openTags) !== false) {
                          // tag found
                          do {
                              // close tags till matching tag found
                              $last_open_tag = array_pop($this-&gt;openTags);
                              $output .= $this-&gt;elements[$last_open_tag]-&gt;htmlAfter;
                          } while ($last_open_tag !== $tag);
                          $allow_child_tags = true;
                          $allow_child_quotes = true;
                      }
                  } else {
                      // opening tag
                      // separate tag name from argument if there is one
                      $equal_pos = strpos($tag_contents, '=');
                      if ($equal_pos === false) {
                          $tag_name = $tag_contents;
                      } else {
                          $tag_name = substr($tag_contents, 0, $equal_pos);
                          $tag_argument = substr($tag_contents, $equal_pos + 1);
                      }
                      if (isset($this-&gt;elements[$tag_name]) === true) {
                          // tag found
                          if (($this-&gt;elements[$tag_name]-&gt;allowParentTags === true || count($this-&gt;openTags) === 0) &amp;&amp; $allow_child_tags === true) {
                              // add tag to open tag list and set flags
                              $this-&gt;openTags[] = $tag_name;
                              $allow_child_tags = $this-&gt;elements[$tag_name]-&gt;allowChildTags;
                              $allow_child_quotes = $this-&gt;elements[$tag_name]-&gt;allowChildQuotes;
                              $output .= $this-&gt;elements[$tag_name]-&gt;htmlBefore;
                              // if argument exists
                              if ($equal_pos !== false) {
                                  if (strpos($tag_argument, '"') === false) {
                                      $output .= $tag_argument;
                                  }
                                  $output .= $this-&gt;elements[$tag_name]-&gt;htmlCenter;
                              }
                          }
                      }
                  }
  
                  $string_segment = strtok('[');
              } while ($string_segment !== false);
              // close left over tags
              while ($tag = array_pop($this-&gt;openTags)) {
                  $output .= $this-&gt;elements[$tag]-&gt;htmlAfter;
              }
              return $output;
          }
      }
  ?&gt;
  
  &lt;?php
  
      class Lev_TextProcessor_Extension_BbCode_Element {
  
          public $htmlBefore;
          public $htmlAfter;
          public $htmlCenter;
          public $allowChildQuotes;
          public $allowChildTags;
          public $allowParentTags;
  
          public function __construct($html_before, $html_after, $html_center = '', $allow_child_quotes = true, $allow_child_tags = true, $allow_parent_tags = true) {
              if ($allow_child_quotes === false &amp;&amp; $allow_child_tags === true) throw new Lev_TextProcessor_Exception('You may not allow child tags if you do not allow child quotes.');
              $this-&gt;htmlBefore = $html_before;
              $this-&gt;htmlAfter = $html_after;
              $this-&gt;htmlCenter = $html_center;
              $this-&gt;allowChildQuotes = $allow_child_quotes;
              $this-&gt;allowChildTags = $allow_child_tags;
              $this-&gt;allowParentTags = $allow_parent_tags;
          }
      }
  ?&gt;
  </code></pre>
  
  <p><strong>edit</strong></p>
  
  <p>Fixed by creating the following class for tokenizing.</p>
  
  <pre><code>&lt;?php
  
      // unlike PHP's strtok() function, this class will not skip over empty tokens.
      class Lev_TextProcessor_Tokenizer {
  
          protected $string;
  
          public function __construct($string) {
              $this-&gt;string = $string;
          }
  
          public function getToken($token) {
              $segment_length = strcspn($this-&gt;string, $token);
              $token = substr($this-&gt;string, 0, $segment_length);
              $this-&gt;string = substr($this-&gt;string, $segment_length + 1);
              return $token;
          }
      }
  ?&gt;
  </code></pre>