void function ( String arg0 , String arg1 , List < String > arg2 ) { arg0 = preprocessField ( arg0 , arg1 ) ; for ( String loc0 : WORD_SPLIT_PATTERN . split ( arg0 ) ) { if ( ! loc0 . isEmpty ( ) ) { arg2 . add ( loc0 . toLowerCase ( ) ) ; } } }
the default implementation normalizes text to lowercase and removes punctuation . this can be subclassed . concode_field_sep String WORD_SPLIT_DEF concode_elem_sep Pattern WORD_SPLIT_PATTERN concode_elem_sep String WORD_SPLIT_PROP concode_field_sep String preprocessField concode_elem_sep String removeHtml