1. A method of generating a language model using non-speech metadata, the method comprising: applying a tree growing algorithm to highly fragmented non-speech metadata associated with a caller to a spoken dialog system, wherein the highly fragmented non-speech metadata does not describe physical characteristics of the caller; identifying projections based on the highly fragmented non-speech metadata and leaf nodes in which a history appears as a result of applying the tree growing algorithm to yield identified projections; and estimating, via a processor of a computing device, a conditional, metadata dependent language model based on the identified projections and speech data from the caller.