relevant keywords for disease would be the actual definition and origin luckily we asked for keywordsso let' pick few more interesting ones such as ndi ndi is lowercased version of ndior "nephrogenic diabetes insipidus,the most common acquired form of diabetes lowercase keywords are returned because that' how they're stored in the index when we put it through the standard analyzer when indexing we didn' specify anything at all while indexingso the standard analyzer was used by default other interesting keywords in the top are avpa gene related to diabetesthirsta symptom of diabetesand amiloridea medication for diabetes these keywords do seem to profile diabetesbut we're missing multi-term keywordswe stored only individual terms in the index because this was the default behavior certain words will never show up on their own because they're not used that often but are still significant when used in combination with other terms currently we miss out on the relationship between certain terms take avpfor exampleif avp were always written in its full form "nephrogenic diabetes insipidus,it wouldn' be picked up storing -grams (combinations of number of wordstakes up storage spaceand using them for queries or aggregations taxes the search server deciding where to stop is balance exercise and depends on your data and use case generallybigrams (combination of two termsare useful because meaningful bigrams exist in the natural languagethough -grams not so much bigram key concepts would be useful for disease profilingbut to create those bigram significant term aggregations you' need them stored as bigrams in your index as is often the case in data scienceyou'll need to go back several steps to make few changes let' go back to the data preparation phase step revisiteddata preparation for disease profiling it shouldn' come as surprise that you're back to data preparationas shown in figure the data science process is an iterative oneafter all when you indexed your datayou did virtually no data cleansing or data transformations you can add data cleansing now byfor instancestop word filtering stop words are words that are so common that they're often discarded because they can pollute the results we won' go into stop word filtering (or other data cleansingherebut feel free to try it yourself to index bigrams you need to create your own token filter and text analyzer token filter is capable of putting transformations on tokens your specific token filter data cleansing examplestop word filtering data preparation data transformation examplelowercasing combining data figure data science process step data preparation data cleansing for text can be stop word filteringdata transformation can be lowercasing of characters