title: XSLT tokenize - capturing the separators
body: <p>here is a piece of code in XSL which tokenizes a text into fragments separated by interpunction and similar characters. I'd like to ask if there is a possibility to somehow capture the strings by which the text was tokenized, for example the comma or dot etc.</p>
  
  <pre><code>&lt;xsl:stylesheet version="2.0" exclude-result-prefixes="xs xdt err fn" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:fn="http://www.w3.org/2005/xpath-functions" xmlns:err="http://www.w3.org/2005/xqt-errors" xmlns:xdt="http://www.w3.org/2005/xpath-datatypes"&gt;
  &lt;xsl:output method="xml" indent="yes"/&gt;
  &lt;xsl:template match="GENERUJ"&gt;
      &lt;TEXT&gt;
          &lt;xsl:variable name="text"&gt;
              &lt;xsl:value-of select="normalize-space(unparsed-text(@filename, 'UTF-8'))" disable-output-escaping="yes"/&gt;
          &lt;/xsl:variable&gt;
          &lt;xsl:for-each select="tokenize($text, '(\s+(&amp;quot;|\(|\[|\{))|((&amp;quot;|,|;|:|\s\-|\)|\]|\})\s+)|((\.|\?|!|;)&amp;quot;?\s*)' )"&gt;
              &lt;xsl:choose&gt;
                  &lt;xsl:when test="string-length(.)&amp;gt;0"&gt;
                      &lt;FRAGMENT&gt;
                          &lt;CONTENT&gt;
                              &lt;xsl:value-of select="."/&gt;
                          &lt;/CONTENT&gt;
                          &lt;LENGTH&gt;
                              &lt;xsl:value-of select="string-length(.)"/&gt;
                          &lt;/LENGTH&gt;
                      &lt;/FRAGMENT&gt;
                  &lt;/xsl:when&gt;
                  &lt;xsl:otherwise&gt;
                      &lt;FRAGMENT_COUNT&gt;
                          &lt;xsl:value-of select="last()-1"/&gt;
                      &lt;/FRAGMENT_COUNT&gt;
                  &lt;/xsl:otherwise&gt;
              &lt;/xsl:choose&gt;
          &lt;/xsl:for-each&gt;
      &lt;/TEXT&gt;
  &lt;/xsl:template&gt;
  </code></pre>
  
  <p></p>
  
  <p>As you see the constructed tags CONTENTS, LENGTH, I'd like to add one called SEPARATOR if you know what I mean. I couldnt find any answer to this on the internet and I'm just a beginner with xsl transformations so I'm looking for a quick solution. Thank you in advance.</p>