InputTokenizer function ( boolean arg0 ) { return new QuotedStringTokenizer ( true , arg0 , false ) ; }
use an input string tokenizer that supports quoted arguments and character escapes . forcing lenient to true makes the following apply : unclosed quotations are treated as a single string from the opening quotation to the end of the arguments rather than throwing an exception concode_field_sep PlaceHolder placeHolder concode_field_sep InputTokenizer rawInput concode_elem_sep InputTokenizer spaceSplitString concode_elem_sep List<SingleArg> tokenize