title: Split string using custom regex java
body: <p>I am building a compiler. Some of the specifications of this are the following:</p>
  
  <ul>
  <li>String literals are enclosed by dollar sign ("$") - eg. $ string sample $</li>
  <li>Comments are enclosed by "*" - eg. * sample comment *</li>
  <li>Comments could exist anywhere execept between operations - eg. 4 + * sample comment * 5 - (this is not allowed)</li>
  </ul>
  
  <p>Now I have to split a source code line to tokenize it.
  Example case:</p>
  
  <pre><code>PRINT $ THE FLOAT IS $ * DISPLAY THE RESULT *
  </code></pre>
  
  <p>As I will tokenize it, it should produce:</p>
  
  <pre><code>PRINT - token is KEYWORD
  THE FLOAT IS - token is STRING_LITERAL
  DISPLAY THE RESULT - token is COMMENT 
  </code></pre>
  
  <p>I would like to know the most efficient way to obtain this. Note that I still have to validate the occurence of string literal and comment. (Ex. Check if it is properly enclosed). So far my way is to split each line by whitespaces and and when a lexeme contains a "$" or "*", I will validate the string literal. Here is my implementation:</p>
  
  <pre><code>private void getLexemes(){
      for(String line : newSourceCode){
          String[] lexemesInALine = line.trim().split("[\\s]+");
          for(String lexemeInALine : lexemesInALine){
              if(!(lexemeInALine.contains("$"))){
                  lexemes.add(lexemeInALine);
                  tempTokens.add(findToken(lexemeInALine));
                  line = line.replaceFirst(lexemeInALine,"").trim();
              }else{
                  validateStringType(line);
                  break;
              }
          }
  </code></pre>
  
  <p>Thank you for the help.</p>