1. A method in a computing system of selectively updating population language models used by language recognition systems, the method comprising: receiving events for representing user changes to local language models, wherein the events are received from devices of a plurality of users; receiving information characterizing the plurality of users, wherein the information includes social-networking friend data for the users; identifying, from the received information, a user cluster based on the social-networking friend data, wherein the user cluster is for representing a subset of users sharing matching or associated instances of the social-networking friend data, and wherein the identifying is performed by a hardware processor; generating or updating a population language model for the user cluster, wherein the generating or updating of the population language model includes: identifying a subset of the events associated with the user cluster for initiating the generation or update of the population language model, and filtering the events by excluding events associated with a blacklist of vocabulary not to be added or events associated with a whitelist of vocabulary not to be deleted; aggregating the user changes and associated words corresponding to the subset of the events; and providing the population language model or updates thereof to a computing device of one or more users in the user cluster for subsequently recognizing input information provided to the computing device by the one or more users in the user cluster.