title: Tokenising a String containing empty tokens
body: <p>I have a seemingly simple problem of splitting a comma separated <code>String</code> into tokens, whereby the output should include empty tokens in cases where:</p>
  
  <ul>
  <li>The first character in the <code>String</code> is a comma.</li>
  <li>The last character in the <code>String</code> is a comma.</li>
  <li>Two consecutive commas occur.</li>
  </ul>
  
  <p>For example, for the <code>String</code>: <code>",abd,def,,ghi,"</code> should yield the output: <code>{"", "abd", "def", "", "ghi", ""}</code>.</p>
  
  <p>I have tried using <code>String.split</code>, <code>Scanner</code> and <code>StringTokenizer</code> for this but each gives a different undesired output (examples below).  Can anyone suggest an <strong>elegant</strong> solution for this, preferably using JDK classes?  Obviously I could code something myself but I feel like I'm missing something on one of the three approaches mentioned.  Note that the delimiter is a fixed <code>String</code> although not necessarily a comma, nor a single character.</p>
  
  <p><strong>Example Code</strong></p>
  
  <pre><code>import java.util.*;
  
  public class Main12 {
    public static void main(String[] args) {
      String s = ",abd,def,,ghi,";
      String[] tokens = s.split(",");
  
      System.err.println("--- String.split Output ---");
      System.err.println(String.format("%s -&gt; %s", s, Arrays.asList(tokens)));
  
      for (int i=0; i&lt;tokens.length; ++i) {
        System.err.println(String.format("tokens[%d] = %s", i, tokens[i]));
      }
  
      System.err.println("--- Scanner Output ---");
  
      Scanner sc = new Scanner(s);
      sc.useDelimiter(",");
      while (sc.hasNext()) {
        System.err.println(sc.next());
      }
  
      System.err.println("--- StringTokenizer Output ---");
  
      StringTokenizer tok = new StringTokenizer(s, ",");
      while (tok.hasMoreTokens()) {
        System.err.println(tok.nextToken());
      }
    }
  }
  </code></pre>
  
  <p><strong>Output</strong></p>
  
  <pre><code>$ java Main12
  --- String.split Output ---
  ,abd,def,,ghi, -&gt; [, abd, def, , ghi]
  tokens[0] =
  tokens[1] = abd
  tokens[2] = def
  tokens[3] =
  tokens[4] = ghi
  --- Scanner Output ---
  abd
  def
  
  ghi
  --- StringTokenizer Output ---
  abd
  def
  ghi
  </code></pre>