title: FullTextIndex in NexusDB, How to tokenize the searchstring
body: <p>We are using NexusDB for a small database. We have a table with a FulltextIndex defined on it.
  The index is configured with the following options:</p>
  
  <ul>
  <li>Character separator</li>
  <li>ccPunctuationDash</li>
  <li>ccPunctuationOther</li>
  </ul>
  
  <p>The user enters a search text in an edit box, and then an SQL statement is constructed with the following WHERE clause (%s substituted with the Editbox.text of course):</p>
  
  <pre><code>WHERE CONTAINS(FullIdx, ''%s'')
  </code></pre>
  
  <p>When the user enters multiple words in the editbox this goes wrong as the two separate words should have been embedded in the WHERE clause like this:</p>
  
  <pre><code>WHERE CONTAINS(FullIdx, 'word1' and 'word2')
  </code></pre>
  
  <p>So i have to parse the textbox value, scan it for spaces and split the text at those points. That made me wonder if it was possible to parse the search text for every setting of the fulltextindex, using the actual definition of the fulltextindex to create the correct where clause.</p>
  
  <p>So if ccPunctuationDash is enabled in the FulltextIndex definition, than the search text is also split on a '-'.</p>
  
  <p>If you think of it, it is exactly the same process as when the index is created and all strings are tokenized ...</p>
  
  <p>My question: what is the easiest way of tokenizing a searchstring according to the settings of a FUlltextIndex?</p>